cleanup attribute, cf. attr(cluster, "cleanup").parallelly::serializedSize().resolved() for ClusterFuture:s would produce Error: 'inherits(future, "Future")' is not TRUE instead of an intended,
informative error message that the connection to the parallel
worker is broken.workers argument. For example,
plan(cluster, workers = cl), where cl is a cluster object,
would come with an extra overhead, because the workers object was
unnecessarily transferred to the cluster nodes.plan(multisession, workers = I(n)), and same for cluster,
preserves the "AsIs" class attribute on the workers argument so
that it is propagated to parallelly::makeClusterWorkers().future.* options.getExpression() on 'cluster' future could under some
circumstances call local() on the global search path rather than
base::local() as intended. For example, if a package that
exports its own local() function was attached, then that would be
called instead, often leading to a hard-to-troubleshoot error.Add prototype of an internal event-logging framework for the purpose of profiling futures and their backends.
Add option future.globalenv.onMisuse for optionally asserting
that a future expression does not result in variables being added
to the global environment.
Add option future.onFutureCondition.keepFuture for controlling
whether FutureCondition objects should keep a copy of the
Future object or not. The default is to keep a copy, but if the
future carries large global objects, then the FutureCondition
will also be large, which can result in memory issues and slow
downs.
The 'multiprocess' strategy, which has been deprecated since future 1.20.0 [2020-10-30] is now defunct. Please use 'multisession' (recommended) or 'multicore' instead.
Add optional assertion of the internal Future state field.
remote(). Note that plan(remote, ...) has been
deprecated since future 1.24.0 [2022-02-19] and defunct since
future 1.30.0 (2022-12-15).Deprecated plan(multiprocess, ...) now equals plan(sequential),
while still producing one warning each time a future is created.
Argument local is defunct and has been removed. Previously only
local = FALSE was defunct.
Remove defunct argument value from all resolve() methods.
Remove defunct functions transparent() and TransparentFuture().
futureOf() used listenv::map(), which is deprecated in
listenv (>= 0.9.0) in favor of listenv::mapping().
Starting with R (>= 4.2.0), the internal function myInternalIP()
no longer detected when an attempted system call failed, resulting
in an obscure error instead of falling back to alternatives. This
was because errors produced by system2() no longer inherits from
class simpleError.
plan(cluster, ..., persistent = TRUE) instead. Note
that persistent = TRUE will eventually also become deprecated and
defunct, but by then we will have an alternative solution
available.R CMD check
may produce "checking for detritus in the temp directory ... NOTE"
and how to avoid them.plan(sequential) and when getOption("warn") == 0.
This bug was introduced in future 1.26.0 [2022-05-27].Using the deprecated plan(multiprocess) will now trigger a
deprecation warning each time a multiprocess future is created.
This means that there could be a lot of warnings produced. Note
that multiprocess has been deprecated since future 1.20.0
[2020-10-30]. Please use multisession (recommended) or
multicore instead.
Removing values(), which has been defunct since future
1.23.0. Use value() instead.
source(..., local = TRUE) is preferred over
source() when used inside futures.do.call(plan, args = list(multisession, workers = 2)) would
ignore the workers argument, and any other arguments.Previously deprecated use of local = FALSE with futures is now
defunct.
The R option to temporarily allow plan(transparent) although it
was declared defunct has now been removed; plan(transparent),
together with functions transparent() and TransparentFuture()
are now formally defunct.
Using argument persistent with multisession futures is now
defunct. Previously only persistent = TRUE was defunct.
Use CSS style to align image to the right instead of non-HTML5
attribute align="right".
Avoid nested <em> tags in HTML-generated help pages.
workers = 1 can now be overridden by
specifying workers = I(1).Some warnings and errors showed the wrong call.
print() for FutureResult would report captured conditions all
with class list, instead of their condition classes.
R CMD check --as-cran on R-devel and MS Windows would
trigger a NOTE on "Check: for detritus in the temp directory" and
"Found the following files/directories: 'Rscript1349cb8aeeba0'
...". There were two package tests that explicitly created PSOCK
cluster without stopping them. A third test launched multisession
future without resolving it, which prevented the PSOCK worker to
terminate. This was not detected in R 4.2.0. It is not a problem
on macOS and Linux, because there background workers are
automatically terminated when the main R session terminates.R options and environment variables are now reset on the workers after future is resolved as they were after any packages required by the future has been loaded and attached. Previously, they were reset to what they were before these were loaded and attached. In addition, only pre-existing R options and environment variables are reset. Any new ones added are not removed for now, because we do not know which added R options or environment variables might have been added from loading a package and that are essential for that package to work.
If it was changed while evaluating the future expression, the current working directory is now reset when the future has been resolved.
futureSessionInfo() gained argument anonymize. If TRUE
(default), host and user names are anonymized.
futureSessionInfo() now also report on the main R session
details.
The bug fix in future 1.22.0 that addressed the issue where
object a in future(fcn(), globals = list(a = 42, fcn = function() a)) would not be found has been redesigned in a more
robust way.
Use of packages such as data.table and ff in cluster and multisession futures broke in future 1.25.0. For data.table, we saw "Error in setalloccol(ans) : verbose must be TRUE or FALSE". For ff, we saw "Error in splitted$path[nopath] <- getOption("fftempdir") : replacement has length zero". See 'Significant Changes' for why and how this was fixed.
The deprecation warning for using local = FALSE was silenced for
sequential futures since future 1.25.0.
futureCall() ignored arguments stdout, conditions,
earlySignal, label, and gc.
Strategy 'transparent' was deprecated in future 1.24.0 and is
now defunct. Use plan(sequential, split = TRUE) instead.
Strategy 'multiprocess' was deprecated in future 1.20.0, and
'remote' was deprecated in future 1.24.0. Since then, attempts
to use them in plan() would produce a deprecation warning, which
was limited to one per R session. Starting with this release, this
warning is now produced whenever using plan() with these
deprecated future strategies.
Now f <- future(..., stdout = structure(TRUE, drop = TRUE)) will
cause the captured standard output to be dropped from the future
object as soon as it has been relayed once, for instance, by
value(f). Similarly, conditions = structure("conditions", drop = TRUE) will drop captured non-error conditions as soon as they have
been relayed. This can help decrease the amount of memory used,
especially if there are many active futures.
Now resolve() respects option future.wait.interval. Previously,
it was hardcoded to poll for results every 0.1 seconds.
value() will only attempt to recover UTF-8 symbols in the
captured standard output if the future was evaluated on an MS
Windows that does not support capturing of UTF-8 symbols. Support
for UTF-8 capturing on also MS Windows was added in R 4.2.0, but
it typically requires an up-to-date MS Windows 10 or MS Windows
Server 2022.future.wait.interval was decreased
from 0.2 seconds to 0.01 seconds. This controls the polling
frequency for finding an available worker when all workers are
currently busy. Starting with this release, this option also
controls the polling frequency of resolve().plan(multicore, workers = 2)
and plan(sequential, split = TRUE) introduced breaking side
effects to the futures evaluated.future(..., seed = TRUE) forwards the RNG state in the
calling R session. Previously, it would leave it intact.plan() and tweak() preserve calls in arguments,
e.g. plan(multisession, workers = 2, rscript_startup = quote(options(socketOptions="no-delay"))), and tweak(..., abc = quote(x == y)).nbrOfFreeWorkers() would produce "Error: 'is.character(name)' is
not TRUE" for plan(multisession, workers = 1).
Internal calls to FutureRegistry(action = "collect-first") and
FutureRegistry(action = "collect-last") could signal errors early
when polling resolved().
Strategy 'remote' is deprecated in favor of 'cluster'. The
plan() function will give an informative deprecation warning when
'remote' is used. For now, this warning is given only once per R
session.
Strategy 'transparent' is deprecated in favor of 'sequential' with
argument split = TRUE set. The plan() function will give an
informative deprecation warning when 'transparent' is used. For
now, this warning is given only once per R session.
plan() now produces a one-time warning if a 'transparent'
strategy is set. The warning reminds the user that 'transparent'
should only be used for troubleshooting purposes and never be used
in production. These days plan(sequential, split = TRUE) together
with debug() is probably a better approach for
troubleshooting. The long-term plan is to deprecate the
'transparent' strategy.
Support for persistent = TRUE with multisession futures is
defunct.
\u2713) would be relayed
as <U+2713> (8 characters). The reason for this is a limitation
in R itself on MS Windows. Now, value() attempts to recover such
MS Windows output to UTF-8 before relaying it. There is an option
for disabling this new feature.quit() must not be used in
forked R processes.future(..., seed) will set the random seed as late as
possible just before the future expression is evaluated.
Previously it was done before package dependencies where attached,
which could lead to non-reproduce random numbers in case a package
dependency would update the RNG seed when attached.values(), which has been deprecated since future 1.20.0, is
now defunct. Use value() instead.
Support for persistent = TRUE with multisession futures is
defunct. If still needed, a temporary workaround is to use cluster
futures. However, it is likely that support for persistent will
eventually be deprecated for all future backends.
Argument value of resolve(), deprecated since future
1.15.0, is defunct in favor of argument result.
parallel::makeCluster(..., type = "FORK").
This test is disabled on macOS, where it appears that the main R
session becomes unstable after the FORK node is terminated.A lazy future remains a generic future until it is launched, which means it is not assigned a future backend class until launched.
Argument seed for futureAssign() and futureCall() now
defaults to FALSE just like for future().
R_FUTURE_* environment variables are now only read when the
future package is loaded, where they set the corresponding
future.* option. Previously, some of these environment variables
were queried by different functions as a fallback to when an option
was not set. By only parsing them when the package is loaded, it
decrease the overhead in functions, and it clarifies that options
can be changed at runtime whereas environment variables should only
be set at startup.
The overhead of initiating futures have been significantly reduced.
For example, the roundtrip time for value(future(NULL)) is about
twice as fast for 'sequential', 'cluster', and 'multisession'
futures. For 'multicore' futures the roundtrip speedup is about
20%. The speedup comes from pre-compiling the R expression that
will be used to resolve the future expression into R expression
templates which then can quickly compiled for each future. This
speeds up the creation of these expression by ~10 times, compared
when re-compiling them each time.
The default timeout for resolved() was decreased from 0.20 to
0.01 seconds for cluster/multisession and multicore futures, which
means they will spend less time waiting for results when they are
not available.
Analogously to how globals may be scanned for "non-exportable"
objects when option future.globals.onReference is set to
"error" or "warning", value() will now check for similar
problems in the resolved value object. An example of this is f <- future(xml2::read_xml("<body></body>")), which will result in an
invalid xml_document object if run in parallel, because such
objects cannot be transferred between R processes.
In addition to specify which condition classes to be captured and
relayed, it is now possible to also specify condition classes to be
ignored. For example, conditions = structure("condition", exclude = "message") captures all conditions but message conditions.
Now cluster futures use homogeneous = NULL as the default instead
of homogeneous = TRUE. The new default will result in the
parallelly package trying to infer whether TRUE or FALSE should
be used based on the workers argument.
Now the the post-mortem analysis report of multicore and cluster futures in case their results could not be retrieved include information on globals and their sizes, and if some of them are non-exportable. A similar, detailed report is also produced when a cluster future fails to set up and launch itself on a parallel worker.
if option future.fork.multithreading.enable is FALSE,
RcppParallel, in addition to OpenMP, is forced to run with
a single threaded whenever running in a forked process
(='multicore' futures). This is done by setting environment
variable RCPP_PARALLEL_NUM_THREADS to 1.
Add futureSessionInfo() to get a quick overview of the future
framework, its current setup, and to run simple tests on it.
Now plan(multicore) warns immediately if multicore processing,
that is, forked processing, is not supported, e.g. when running in
the RStudio Console.
plan(multiprocess, workers = n) did not warn about 'multiprocess'
being deprecated when argument workers was specified.
getGlobalsAndPackages() could throw a false error on "Did you
mean to create the future within a function? Invalid future
expression tries to use global ... variables that do not exist:
... is solely part of a formula or used
in some S4 generic functions.
When enabled, option future.globals.onReference could falsely
alert on 'Detected a non-exportable reference (externalptr) in one
of the globals (<unknown>) used in the future expression' in
globals, e.g. when using future.apply or furrr map-reduce
functions when using a 'multisession' backend.
future(fcn(), globals = list(a = 42, fcn = function() a)) would
fail with "Error in fcn() : object 'a' not found" when using
sequential or multicore futures. This affected also map-reduce
calls such as future.apply::future_lapply(1, function(x) a, future.globals = list(a = 42)).
Resolving a 'sequential' future without globals would result in
internal several ...future.* objects being written to the calling
environment, which might be the global environment.
Environment variable R_FUTURE_PLAN would propagate down with
nested futures, forcing itself onto also nested future plans. Now
it is unset in nested futures, resulting in a sequential future
strategy unless another was explicitly set by plan().
Transparent futures no longer warn about local = FALSE being
deprecated. Although local = FALSE is being deprecated, it is
still used internally by 'transparent' futures for a while longer.
Please do not use 'transparent' futures in production code and
never in a package.
remote() could produce an error on "object 'homogeneous' not
found".
nbrOfFreeWorkers() for 'cluster' futures assumed that the current
plan is set to cluster too.
In order to handle them conditionally higher up in the call chain,
warnings and errors produced from using the random number generator
(RNG) in a future without declaring the intention to use one are
now of class RngFutureWarning and RngFutureError, respectively.
Both of these classes inherits from RngFutureCondition.
Now run-time errors from resolving a future take precedence over
RngFutureError:s. That is, future({ rnorm(1); log("a") }, seed = FALSE) will signal an error 'log("a")' instead of an RNG error
when option future.rng.onMisuse is set to "error".
nbrOfFreeWorkers() to query how many workers are free to take
on futures immediately. Until all third-party future backends have
implemented this, some backends might produce an error saying it is
not yet supported.future(..., seed = TRUE) with 'sequential' futures would set the
RNG kind of the parent process. Now it behaves the same regardless
of future backend.
Signaling immediateCondition:s with 'multicore' could result in
Error in save_rds(obj, file) : save_rds() failed to rename temporary save file '/tmp/RtmpxNyIyK/progression21f3f31eadc.rds.tmp' (NA bytes; last modified on NA) to '/tmp/RtmpxNyIyK/progression21f3f31eadc.rds' (NA bytes; last modified on NA). There was an assertion at the end of
the internal save_rds() function that incorrectly assumed that
the target file should exist. However, the file might have already
been processed and removed by the future in the main R session.
value() with both a run-time error and an RNG mistake would
signal the RNG warning instead of the run-time error when the
for-internal-use-only argument signal was set to FALSE.
Due to a mistake introduced in future 1.20.0, the package would
end up assigning a .packageVersion object to the global
environment when loaded.
future::plan("multisession") would produce 'Error in if (debug)
mdebug("covr::package_coverage() workaround ...") : argument is not
interpretable as logical' if and only if the covr package was
loaded.Strategy 'multiprocess' is deprecated in favor of either
'multisession' or 'multicore', depending on operating system and R
setup. The plan() function will give an informative deprecation
warning when 'multiprocess' is used. This warning is given only
once per R session.
Launching R or Rscript with command-line option --parallel=n,
where n > 1, will now use 'multisession' as future strategy.
Previously, it would use 'multiprocess', which is now deprecated.
Support for local = FALSE is deprecated. For the time being, it
remains supported for 'transparent' futures and 'cluster' futures
that use persistent = TRUE. However, note that persistent = TRUE will also deprecated at some point in the future. These
deprecations are required in order to further standardize the
Future API across various types of parallel backends.
Now multisession workers inherit the package library path from the
main R session when they are created, that is, when calling
plan(multisession). To avoid this, use plan(multisession, rscript_libs = NULL), which is an argument passed down to
makeClusterPSOCK(). With this update, 'sequential',
'multisession', and 'multicore' futures see the exact same library
path.
Several functions for managing parallel-style processing have
been moved to a new parallelly package. Specifically,
functions availableCores(), availableWorkers(),
supportsMulticore(), as.cluster(), autoStopCluster(),
makeClusterMPI(), makeClusterPSOCK(), and makeNodePSOCK()
have been moved. None of them are specific to futures per se and
are likely useful elsewhere too. Also, having them in a separate,
standalone package will speed up the process of releasing any
updates to these functions. The code base of the future
package shrunk about 10-15% from this migration. For backward
compatibility, the migrated functions remain in this package as
re-exports.
Setting up a future strategy with argument split = TRUE will
cause the standard output and non-error conditions to be split
("tee:d") on the worker's end, while still relaying back to the
main R session as before. This can be useful when debugging with
browse() or debug(), e.g. plan(sequential, split = TRUE).
Without it, debug output is not displayed.
Now multicore futures relay immediateCondition:s in a near-live
fashion.
It is now possible to pass any arguments that makeClusterPSOCK()
accepts in the call to plan(cluster, ...) and plan(multisession, ...). For instance, to set the working directory of the cluster
workers to a temporary folder, pass argument rscript_startup = "setwd(tempdir())". Another example is rscript_libs = c(libs, "*") to prepend the library path on the worker with the paths in
libs.
plan() and tweak() check for even more arguments that must not
be set by either of them. Specifically, attempts to adjust the
following arguments of future() will result in an error:
conditions, envir, globals, packages, stdout, and
substitute in addition to already validated lazy and seed.
tweak() now returns a wrapper function that calls the original future
strategy function with the modified defaults. Previously, it would make a
copy of the original function with modified argument defaults. This new
approach will make it possible to introduce new future arguments that can
be modified by tweak() and plan() without having to update every future
backend package, e.g. the new split = TRUE argument.
Add a 'Best Practices for Package Developers' vignette.
Add a 'How the Future Framework is Validated' vignette.
substitute = TRUE.Since last version, future 1.19.1, future(..., conditions = character(0L)) would no longer avoid intercepting conditions as
intended; instead, it muffles all conditions. From now on, use
conditions = NULL.
Relaying of immediateCondition:s was not near-live for
multisession and cluster if the underlying PSOCK cluster used
useXDR=FALSE for communication.
print() for Future would also print any attributes of its
environment.
The error message produced by nbrOfWorkers() was incomplete.
Renamed environment variable R_FUTURE_MAKENODEPSOCK_tries used by
makeClusterPSOCK() to R_FUTURE_MAKENODEPSOCK_TRIES.
The Mandelbrot demo would produce random numbers without declaring so.
Strategy 'multiprocess' is deprecated in favor of either 'multisession' or 'multicore', depending on operating system and R setup.
values() is deprecated. Use value() instead.
All backward compatible code for the legacy, defunct, internal
Future element value is now removed. Using or relying on it is
an error.
... as a globals, rather than via arguments, in
higher-level map-reduce APIs such as future.apply and
furrr, arguments in ... could produce an error on "unused
argument".Futures detect when random number generation (RNG) was used to
resolve them. If a future uses RNG without parallel RNG was
requested, then an informative warning is produced. To request
parallel RNG, specify argument seed, e.g. f <- future(rnorm(3), seed = TRUE) or y %<-% { rnorm(3) } %seed% TRUE. Higher-level
map-reduce APIs provide similarly named "seed" arguments to achieve
the same. To, escalate these warning to errors, set option
future.rng.onMisuse to "error". To silence them, set it to
"ignore".
Now, all non-captured conditions are muffled, if possible. For
instance, future(warning("boom"), conditions = c("message")) will
truly muffle the warning regardless of backend used. This was
needed to fix below bug.
makeClusterPSOCK() will now retry to create a cluster node up to
tries (default: 3) times before giving up. If argument port
species more than one port (e.g. port = "random") then it will
also attempt find a valid random port up to tries times before
giving up. The pre-validation of the random port is only supported
in R (>= 4.0.0) and skipped otherwise.
makeClusterPSOCK() skips shell quoting of the elements in
rscript if it inherits from AsIs.
makeClusterPSOCK(), or actually makeNodePSOCK(), gained
argument quiet, which can be used to silence output produced by
manual = TRUE.
If multithreading is disabled but multicore futures fail to
acknowledge the setting on the current system, then an informative
FutureWarning is produced by such futures.
Now availableCores() better supports Slurm. Specifically, if
environment variable SLURM_CPUS_PER_TASK is not set, which
requires that option --slurm-cpus-per-task=n is specified and
SLURM_JOB_NUM_NODES=1, then it falls back to using
SLURM_CPUS_ON_NODE, e.g. when using --ntasks=n.
Now availableCores() and availableWorkers() supports
LSF/OpenLava. Specifically, they acknowledge environment variable
LSB_DJOB_NUMPROC and LSB_HOSTS, respectively.
plan(multisession), plan(cluster, workers = <number>), and
makeClusterPSOCK() which they both use internally, sets up
localhost workers twice as fast compared to versions since
future 1.12.0, which brings it back to par with a bare-bone
parallel::makeCluster(..., setup_strategy = "sequential") setup.
The slowdown was introduced in future 1.12.0 (2019-03-07) when
protection against leaving stray R processes behind from failed
worker startup was implemented. This protection now makes use of
memoization for speedup.Sequential and multicore backends, but not multisession, would
produce errors on "'...' used in an incorrect context" in cases
where ... was part of argument globals and not the evaluation
environment.
Contrary to other future backends, any conditions produced while
resolving a sequential future using future(..., conditions = character()) would be signaled, although the most reasonable
expectation would be that they are silenced. Now, all non-captured
conditions are muffled, if possible.
Option future.rng.onMisuse was not passed down to nested futures.
Disabling multithreading in forked processes by setting R option
future.fork.multithreading.enable or environment variable
R_FUTURE_FORK_MULTITHREADING_ENABLE to FALSE would cause
multicore futures to always return value 1L. This bug was
introduced in future 1.17.0 (2020-04-17).
getGlobalsAndPackages() did not always return a globals element
that was of class FutureGlobals.
getGlobalsAndPackages(..., globals) would recalculate
total_size even when it was already calculated or known to be
zero.
getGlobalsAndPackages(Formula::Formula(~ x)) would produce "the
condition has length > 1" warnings (which will become errors in
future R versions).
persistent = TRUE with multisession futures is
deprecated.print() on RichSOCKcluster gives information not only on the
name of the host but also on the version of R and the platform of
each node ("worker"), e.g. "Socket cluster with 3 nodes where 2
nodes are on host 'localhost' (R version 4.0.0 (2020-04-24),
platform x86_64-w64-mingw32), 1 node is on host 'n3' (R version
3.6.3 (2020-02-29), platform x86_64-pc-linux-gnu)".
Error messages from cluster future failures are now more
informative than "Unexpected result (of class 'NULL' !=
'FutureResult')". For example, if the future package is not
installed on the worker, then the error message clearly says so.
Even, if there is an unexpected result error from a PSOCK cluster
future, then the error produced give extra information on node
where it failed, e.g. "Unexpected result (of class 'NULL' !=
'FutureResult') retrieved for ClusterFuture future (label =
'ClusterFuture worker ('RichSOCKnode' #1 on host 'n3' (R
version 3.6.3 (2020-02-29), platform x86_64-pc-linux-gnu)) is out
of sync."
It is now possible to set environment variables on workers before
they are launched by makeClusterPSOCK() by specify them as as
"<name>=<value>" as part of the rscript vector argument,
e.g. rscript = c("ABC=123", "DEF='hello world'", "Rscript"). This
works because elements in rscript that match regular expression
[[:alpha:]_][[:alnum:]_]*=.* are no longer shell quoted.
makeClusterPSOCK() now returns a cluster that in addition to
inheriting from SOCKcluster it will also inherit from
RichSOCKcluster.
Made makeClusterPSOCK() and makeNodePSOCK() agile to the name
change from parallel:::.slaveRSOCK() to parallel:::.workRSOCK()
in R (>= 4.1.0).
makeClusterPSOCK(..., rscript) will not try to locate
rscript[1] if argument homogeneous is FALSE (or inferred to be
FALSE).
makeClusterPSOCK(..., rscript_envs) would result in a syntax
error when starting the workers due to non-ASCII quotation marks if
option useFancyQuotes was not set to FALSE.
plan(list(...)) would produce 'Error in UseMethod("tweak") : no
applicable method for 'tweak' applied to an object of class "list"'
if a non-function object named 'list' was on the search path.
plan(x$abc) with x <- list(abc = sequential) would produce 'Error
in UseMethod("tweak") : no applicable method for 'tweak' applied to
an object of class "c('FutureStrategyList', 'list')"'.
TESTS: R_FUTURE_FORK_ENABLE=false R CMD check ... would produce
'Error: connections left open: ...' when checking the
'multiprocess' example.
Support for persistent = TRUE with multisession futures is
deprecated. If still needed, a temporary workaround is to use
cluster futures. However, it is likely that support for
persistent will eventually be deprecated for all future backends.
Options future.globals.method, future.globals.onMissing, and
future.globals.resolve are deprecated and produce warnings if
set. They may only be used for troubleshooting purposes because
they may affect how futures are evaluated, which means that
reproducibility cannot be guaranteed elsewhere.
values() to value() to clean up and simplify the API.makeClusterPSOCK() gained argument rscript_envs for setting
environment variables in workers on startup, e.g. rscript_envs = c(FOO = "3.14", "BAR").
Now the result of a future holds session details in case an error occurred while evaluating the future.
_R_CHECK_LIMIT_CORES_ set. To better
emulate CRAN submission checks, the future package will, when
loaded, set this environment variable to 'TRUE' if unset and if R CMD check is running. Note that future::availableCores()
respects _R_CHECK_LIMIT_CORES_ and returns at most 2L (two
cores) if detected.Any globals named version and has_future would be overwritten
with "garbage" values internally.
Disabling of multi-threading when using 'multicore' futures did not work on all platforms.
values() S3 methods have been renamed to value() since they
are closely related to the original purpose value(). The
values() methods will continue to work but will soon be formally
deprecated and later be made defunct and finally be removed.
Please replace all values() with value() calls.oplan <- plan(new_strategy) returns the list of all nested
strategies previously set, instead of just the strategy on top of
this stack. This makes it easier to temporarily use another
plan. For the old behavior, use oplan <- plan(new_strategy)[[1]].Now value() detects if a future(..., seed = FALSE) call
generated random numbers, which then might give unreliable results
because non-parallel safe, non-statistically sound random number
generation (RNG) was used. If option future.rng.onMisuse is
"warning", a warning is produced. If "error", an error is
produced. If "ignore" (default), the mistake is silently
ignored. Using seed = NULL is like seed = FALSE but without
performing the RNG validation.
For convenience, argument seed of future() may now also be an
ordinary single integer random seed. If so, a L'Ecuyer-CMRG RNG
seed is created from this seed. If seed = TRUE, then a
L'Ecuyer-CMRG RNG seed based on the current RNG state is used. Use
seed = FALSE when it is known that the future does not use RNG.
ClusterFuture:s now relay immediateCondition:s back to the main
process momentarily after they are signaled and before the future
is resolved.
future.fork.multithreading.enable or environment variable
R_FUTURE_FORK_MULTITHREADING_ENABLE to FALSE. This requires
that RhpcBLASctl package is installed. Parallelization via
multi-threaded processing (done in native code by some packages and
externally library) while at the same time using forked (aka
"multicore") parallel processing is unstable in some cases. Note
that this is not only true when using plan(multicore) but also
when using, for instance, parallel::mclapply(). This is in beta
so the above names and options might change later.Evaluation of futures could fail if the global environment
contained functions with the same names as a small set of base R
functions, e.g. raw(), list(), and options().
future(alist(a =)) would produce "Error in objectSize_list(x,
depth = depth - 1L) : argument "x_kk" is missing, with no default"
Future and FutureResult objects with an internal version 1.7 or
older have been deprecated since 1.14.0 (July 2019) and are now
defunct.
Defunct hidden argument progress of resolve(), and hidden
arguments/fields condition and calls of FutureResult are now
gone.
makeClusterPSOCK() draws a random
port from (when argument port is not specified) can now be
controlled by environment variable R_FUTURE_RANDOM_PORTS. The
default range is still 11000:11999 as with the parallel
package.resolved() in future 1.15.0 would
cause lazy futures to block if all workers were occupied.resolved() will now launch lazy futures.Now the "visibility" of future values is recorded and reflected by
value().
Now option future.globals.onReference defaults to environment
variable R_FUTURE_GLOBALS_ONREFERENCE.
?makeClusterPSOCK with
instructions on how to troubleshoot when the setup of local and
remote clusters fail.values() would resignal immediateCondition:s despite those
should only be signaled at most once per future.
makeClusterPSOCK() could produce warnings like "cannot open file
'/tmp/alice/Rtmpi69yYF/future.parent=2622.a3e32bc6af7.pid': No such
file", e.g. when launching R workers running in Docker containers.
Package would set or update the RNG state of R (.Random.seed)
when loaded, which could affect RNG reproducibility.
Package could set .Random.seed to NULL, instead of removing it,
which in turn would produce a warning on "'.Random.seed' is not an
integer vector but of type 'NULL', so ignored" when the next random
number generated.
Now a future assignment to list environments produce more informative error messages if attempting to assign to more than one element.
makeClusterMPI() did not work for MPI clusters with comm other
than 1.
Argument value of resolve() is deprecated. Use result
instead.
Use of internal argument evaluator to future() is now defunct.
All types of conditions are now captured and relayed. Previously,
only conditions of class message and warning were relayed.
If one of the futures in a collection produces an error, then
values() will signal that error as soon as it is detected. This
means that while calling values() guarantees to resolve all
futures, it does not guarantee that the result from all futures are
gathered back to the master R session before the error is relayed.
values() now relays stdout and signal as soon as possible as
long as the standard output and the conditions are relayed in their
original order.
If a captured condition can be "muffled", then it will be muffled.
This helps to prevent conditions from being handled twice by
condition handlers when futures are evaluated in the main R
session, e.g. plan(sequential). Messages and warnings were
already muffled in the past.
Forked processing is considered unstable when running R from
certain environments, such as the RStudio environment. Because of
this, 'multicore' futures have been disabled in those cases since
future 1.13.0. This change caught several RStudio users by
surprise. Starting with future 1.14.0, an informative
one-time-per-session warning will be produced when attempts to use
'multicore' is made in non-supported environments such as RStudio.
This warning will also be produced when using 'multiprocess', which
will fall back to using 'multisession' futures. The warning can be
disabled by setting R option future.supportsMulticore.unstable,
or environment variable FUTURE_SUPPORTSMULTICORE_UNSTABLE to
"quiet".
Now option future.startup.script falls back to environment
variable R_FUTURE_STARTUP_SCRIPT.
Conditions inheriting immediateCondition are signaled as soon as
possible. Contrary to other types of conditions, these will be
signaled only once per future, despite being collected.
Early signaling did not take place for resolved() for
ClusterFuture and MulticoreFuture.
When early signaling was enabled, functions such as resolved() and
resolve() would relay captured conditions multiple times. This would,
for instance, result in the same messages and warnings being outputted
more than once. Now it is only value() that will resignal conditions.
The validation of connections failed to detect when the connection
had been serialized (= a NIL external pointer) on some macOS
systems.
Argument progress of resolve() is now defunct (was deprecated
since future 1.12.0). Option future.progress is ignored.
This will make room for other progress-update mechanisms that are
in the works.
Usage of internal argument evaluator to future() is now
deprecated.
Removed defunct argument output from FutureError().
FutureResult fields/arguments condition and calls are now
defunct. Use conditions instead.
Future and FutureResult objects with an internal version 1.7 or
older are deprecated and will eventually become defunct. Future
backends that implement their own Future classes should update to
implement a result() method instead of a value() method for
their Future classes. All future backends available on CRAN and
Bioconductor have already been updated accordingly.
help("supportsMulticore") for more details, e.g. how to
re-enable process forking. Note that parallelization via
'multisession' is unaffected and will still work as before. Also,
when forked processing is disabled, or otherwise not supported,
using plan("multiprocess") will fall back to using 'multisession'
futures.Forked processing can be disabled by setting R option
future.fork.enable to FALSE (or environment variable
R_FUTURE_FORK_ENABLE=false). When disabled, 'multicore' futures
fall back to a 'sequential' futures even if the operating system
supports process forking. If set of TRUE, 'multicore' will not
fall back to 'sequential'. If NA, or not set (the default), a set
of best-practices rules will decide whether forking is enabled or
not. See help("supportsMulticore") for more details.
Now availableCores() also recognizes PBS environment variable
NCPUS, because the PBSPro scheduler does not set PBS_NUM_PPN.
If, option future.availableCores.custom is set to a function,
then availableCores() will call that function and interpret its
value as number of cores. Analogously, option
future.availableWorkers.custom can be used to specify a hostnames
of a set of workers that availableWorkers() sees. These new
options provide a mechanism for anyone to customize
availableCores() and availableWorkers() in case they do not
(yet) recognize, say, environment variables that are specific the
user's compute environment or HPC scheduler.
makeClusterPSOCK() gained support for argument rscript_startup
for evaluating one or more R expressions in the background R worker
prior to the worker event loop launching. This provides a more
convenient approach than having to use, say, rscript_args = c("-e", sQuote(code)).
makeClusterPSOCK() gained support for argument rscript_libs to
control the R package library search path on the workers. For
example, to prepend the folder ~/R-libs on the workers, use
rscript_libs = c("~/R-libs", "*"), where "*" will be resolved
to the current .libPaths() on the workers.
Debug messages are now prepended with a timestamp.
makeClusterPSOCK() did not shell quote the Rscript executable
when running its pre-tests checking whether localhost Rscript
processes can be killed by their PIDs or not.value of resolve() has been renamed to result to
better reflect that not only values are collected when this
argument is used. Argument value still works for backward
compatibility, but will eventually be formally deprecated and then
defunct.If makeClusterPSOCK() fails to create one of many nodes, then it
will attempt to stop any nodes that were successfully created.
This lowers the risk for leaving R worker processes behind.
Future results now hold the timestamps when the evaluation of the future started and finished.
Functions no longer produce "partial match of 'condition' to
'conditions'" warnings with options(warnPartialMatchDollar = TRUE).
When future infix operators (%conditions%, %globals%,
%label%, %lazy%, %packages%, %seed%, and %stdout%) that
are intended for future assignments were used in the wrong context,
they would incorrectly be applied to the next future created. Now
they're discarded.
makeClusterPSOCK() in future (>= 1.11.1) produced warnings
when argument rscript had length(rscript) > 1.
Validation of L'Ecuyer-CMRG RNG seeds failed in recent R devel.
With options(OutDec = ","), the default value of several argument
would resolve to NA_real_ rather than a numeric value resulting
in errors such as "is.finite(alpha) is not TRUE".
Argument progress of resolve() is now deprecated.
Argument output of FutureError() is now defunct.
FutureError no longer inherits simpleError.
makeClusterPSOCK() fails to connect to a worker, it produces
an error with detailed information on what could have happened. In
rare cases, another error could be produced when generating the
information on what the workers PID is.The defaults of several arguments of makeClusterPSOCK() and
makeNodePSOCK() can now be controlled via environment variables
in addition to R options that was supported in the past. An
advantage of using environment variables is that they will be
inherited by child processes, also nested ones.
The printing of future plans is now less verbose when the workers
argument is a complex object such as a PSOCK cluster object.
Previously, the output would include verbose output of attributes,
etc.
R CMD check is running or not. If it is, then a few future-specific
environment variables are adjusted such that the tests play nice
with the testing environment. For instance, it sets the socket
connection timeout for PSOCK cluster workers to 120 seconds
(instead of the default 30 days!). This will lower the risk for
more and more zombie worker processes cluttering up the test
machine (e.g. CRAN servers) in case a worker process is left behind
despite the main R processes is terminated. Note that these
adjustments are applied automatically to the checks of any package
that depends on, or imports, the future package.makeClusterPSOCK() would fail to connect to a worker,
for instance due to a port clash, then it would leave the R worker
process running - also after the main R process terminated. When
the worker is running on the same machine, makeClusterPSOCK()
will now attempt to kill such stray R processes. Note that
parallel::makePSOCKcluster() still has this problem.The future call stack ("traceback") is now recorded when the
evaluation of a future produces an error. Use backtrace() on the
future to retrieve it.
Now futureCall() defaults to args = list() making is easier to
call functions that do not take arguments,
e.g. futureCall(function() 42).
plan() gained argument .skip = FALSE. When TRUE, setting the
same future strategy as already set will be skipped, e.g. calling
plan(multisession) consecutively will have the same effect as
calling it just once.
makeClusterPSOCK() produces more informative error messages
whenever the setup of R workers fails. Also, its verbose messages
are now prefixed with [local output] to help distinguish the
output produced by the current R session from that produced by
background workers.
It is now possible to specify what type of SSH clients
makeClusterPSOCK() automatically searches for and in what order,
e.g. rshcmd = c("<rstudio-ssh>", "<putty-plink>").
Now makeClusterPSOCK() preserves the global RNG state
(.Random.seed) also when it draws a random port number.
makeClusterPSOCK() gained argument rshlogfile.
Cluster futures provide more informative error messages when the communication with the worker node is out of sync.
Argument stdout was forced to TRUE when using single-core
multicore or single-core multisession futures.
When evaluated in a local environment, futureCall(..., globals = "a") would set the value of global a to NULL, regardless if it
exists or not and what its true value is.
makeClusterPSOCK(..., rscript = "my_r") would in some cases fail
to find the intended my_r executable.
ROBUSTNESS: A cluster future, including a multisession one, could retrieve results from the wrong workers if a new set of cluster workers had been set up after the future was created/launched but before the results were retrieved. This could happen because connections in R are indexed solely by integers which are recycled when old connections are closed and new ones are created. Now cluster futures assert that the connections to the workers are valid, and if not, an informative error message is produced.
Calling result() on a non-resolved UniprocessFuture would
signal evaluation errors.
future::future_lapply(). Please use the one in
the future.apply package instead.Add support for manually specifying globals in addition to those
that are automatically identified via argument globals or
%globals%. Two examples are globals = structure(TRUE, add = list(a = 42L, b = 3.14)) and globals = structure(TRUE, add = c("a", "b")). Analogously, attribute ignore can be used to
exclude automatically identified globals.
The error reported when failing to retrieve the results of a future evaluated on a localhost cluster/multisession worker or a forked/multicore worker is now more informative. Specifically, it mentions whether the worker process is still alive or not.
Add makeClusterMPI(n) for creating MPI-based clusters of a
similar kind as parallel::makeCluster(n, type = "MPI") but that
also attempts to workaround issues where parallel::stopCluster()
causes R to stall.
makeClusterPSOCK() and makeClusterMPI() gained argument
autoStop for controlling whether the cluster should be
automatically stopped when garbage collected or not.
BETA: Now resolved() for ClusterFuture is non-blocking also for
clusters of type MPIcluster as created by
parallel::makeCluster(..., type = "MPI").
plan(multiprocess) would not initiate the
workers. Instead workers would be set up only when the first future
was created.value() is
called. This new behavior can be controlled by the argument
stdout to future() or by specifying the %stdout% operator if
a future assignment is used.R option width is passed down so that standard output is captured
consistently across workers and consistently with the master
process.
Now more future.* options are passed down so that they are also
acknowledged when using nested futures.
Add vignette on 'Outputting Text'.
CLEANUP: Only the core parts of the API are now listed in the help
index. This was done to clarify the Future API. Help for non-core
parts are still via cross references in the indexed API as well via
help().
When using forced, nested 'multicore' parallel processing, such as,
plan(list(tweak(multicore, workers = 2), tweak(multicore, workers = 2))), then the child process would attempt to resolve futures
owned by the parent process resulting in an error (on 'bad error
message').
When using plan(multicore), if a forked worker would terminate
unexpectedly, it could corrupt the master R session such that any
further attempts of using forked workers would fail. A forked
worker could be terminated this way if the user pressed
Ctrl-C (the worker receives a SIGINT signal).
makeClusterPSOCK() produced a warning when environment variable
R_PARALLEL_PORT was set to random (e.g. as on CRAN).
Printing a plan() could produce an error when the deparsed call
used to set up the plan() was longer than 60 characters.
future::future_lapply() is defunct (gives an error if called).
Please use the one in the future.apply package instead.
Argument output of FutureError() is formally deprecated.
Removed all FutureEvaluationCondition classes and related
methods.
getGlobalsAndPackages() gained argument maxSize.
makeClusterPSOCK() now produces a more informative warning if
environment variable R_PARALLEL_PORT specifies a non-numeric
port.
Now plan() gives a more informative error message in case it
fails, e.g. when the internal future validation fails and why.
Added UnexpectedFutureResultError to be used by backends for
signaling in a standard way that an unexpected result was retrieved
from a worker.
When the communication between an asynchronous future and a background R process failed, further querying of the future state/results could end up in an infinite waiting loop. Now the failed communication error is recorded and re-signaled if any further querying attempts.
Internal, seldom used myExternalIP() failed to recognize IPv4 answers
from some of the lookup servers. This could in turn produce another error.
In R (>= 3.5.0), multicore futures would produce multiple warnings originating from querying whether background processes have completed or not. These warnings are now suppressed.
More errors related to orchestration of futures are of class
FutureError to make it easier to distinguish them from future
evaluation errors.
Add support for a richer set of results returned by resolved
futures. Previously only the value of the future expression, which
could be a captured error to be resignaled, was expected. Now a
FutureResult object may be returned instead. Although not
supported in this release, this update opens up for reporting on
additional information from the evaluation of futures,
e.g. captured output, timing and memory benchmarks, etc. Before
that can take place, existing future backend packages will have to
be updated accordingly.
backtrace() returns only the last call that produced the error.
It is unfortunately not possible to capture the call stack that led
up to the error when evaluating a future expression.
value() for MulticoreFuture would not produce an error when a
(forked) background R workers would terminate before the future
expression is resolved. This was a limitation inherited from the
parallel package. Now an informative FutureError message is
produced.
value() for MulticoreFuture would not signal errors unless they
inherited from simpleError - now it's enough for them to inherits
from error.
value() for ClusterFuture no longer produces a
FutureEvaluationError, but FutureError, if the connection to
the R worker has changed (which happens if something as drastic as
closeAllConnections() have been called.)
futureCall(..., globals = FALSE) would produce "Error: second
argument must be a list", because the explicit arguments where not
exported. This could also happen when specifying globals by name
or as a named list.
Nested futures were too conservative in requiring global variables to exist, even when they were false positives.
future::future_lapply() is formally deprecated. Please use the
one in the future.apply package instead.
Recently introduced FutureEvaluationCondition classes are
deprecated, because they no longer serve a purpose since future
evaluation conditions are now signaled as-is.
future_lapply() has moved to the future.apply package
available on CRAN.Argument workers of future strategies may now also be a function,
which is called without argument when the future strategy is set up
and used as is. For instance, plan(multiprocess, workers = halfCores) where halfCores <- function() { max(1, round(availableCores() / 2)) } will use half of the number of
available cores. This is useful when using nested future
strategies with remote machines.
On Windows, makeClusterPSOCK(), and therefore
plan(multisession) and plan(multiprocess), will use the SSH
client distributed with RStudio as a fallback if neither ssh nor
plink is available on the system PATH.
Now plan() makes sure that nbrOfWorkers() will work for the new
strategy. This will help catch mistakes such as plan(cluster, workers = cl) where cl is a basic R list rather than a cluster
list early on.
Added %packages% to explicitly control packages to be attached
when a future is resolved, e.g. y %<-% { YT[2] } %packages% "data.table". Note, this is only needed in cases where the
automatic identification of global and package dependencies is not
sufficient.
Added condition classes FutureCondition, FutureMessage,
FutureWarning, and FutureError representing conditions that
occur while a future is setup, launched, queried, or retrieved.
They do not represent conditions that occur while evaluating the
future expression. For those conditions, new classes
FutureEvaluationCondition, FutureEvaulationMessage,
FutureEvaluationWarning, and FutureEvaluationError exists.
if (runif(1) < 1/2) x <- 0; y <- 2 * x.externalptr) can not be
exported, but there are exceptions. By setting options
future.globals.onReference to "warning", a warning is produced
informing the user about potential problems. If "error", an
error is produced. Because there might be false positive, the
default is "ignore", which will cause above scans to be skipped.
If there are non-exportable globals and these tests are skipped, a
run-time error may be produced only when the future expression is
evaluated.The total size of global variables was overestimated, and dramatically so if defined in the global environment and there were are large objects there too. This would sometimes result in a false error saying that the total size is larger than the allowed limit.
An assignment such as x <- x + 1 where the left-hand side (LHS)
x is a global failed to identify x as a global because the
right-hand side (RHS) x would override it as a local variable.
Updates to the globals package fixed this problem.
makeClusterPSOCK(..., renice = 19) would launch each PSOCK worker
via nice +19 resulting in the error "nice: '+19': No such file or
directory". This bug was inherited from
parallel::makePSOCKcluster(). Now using nice --adjustment=19
instead.
Protection against passing future objects to other futures did not work for future strategy 'multicore'.
future_lapply() has moved to the new future.apply package
available on CRAN. The future::future_lapply() function will
soon be deprecated, then defunct, and eventually be removed from
the future package. Please update your code to make use of
future.apply::future_lapply() instead.
Dropped defunct 'eager' and 'lazy' futures; use 'sequential' instead.
Dropped defunct arguments cluster and maxCores; use workers
instead.
In previous version of the future package the FutureError
class was used to represent both orchestration errors (now
FutureError) and evaluation errors (now FutureEvaluationError).
Any usage of class FutureError for the latter type of errors is
deprecated and should be updated to FutureEvaluationError.
Now plan() accepts also strings such as "future::cluster".
Now backtrace(x[[ER]]) works also for non-environment x:s,
e.g. lists.
When measuring the size of globals by scanning their content, for certain types of classes the inferred lengths of these objects were incorrect causing internal subset out-of-range issues.
print() for Future would output one global per line instead of
concatenating the information with commas.
getGlobalsAndPackages().future_lapply() would give "Error in objectSize.env(x, depth =
depth - 1L): object 'nnn' not found" when for instance 'nnn' is
part of an unresolved expression that is an argument value.tweak(), and hence plan(), generates a more informative
error message if a non-future function is specified by mistake,
e.g. calling plan(cluster) with the survival package attached
after future is equivalent to calling plan(survival::cluster)
when plan(future::cluster) was intended.nbrOfWorkers() gave an error with plan(remote). Fixed by making
the 'remote' future inherit cluster (as it should).quit(), but that appeared to have
corrupted the main R session when running on Solaris.Formally defunct 'eager' and 'lazy' futures; use 'sequential' instead.
Dropped previously defunct %<=% and %=>% operators.
makeClusterPSOCK() now defaults to use the Windows PuTTY
software's SSH client plink -ssh, if ssh is not found.
Argument homogeneous of makeNodePSOCK(), a helper function of
makeClusterPSOCK(), will default to FALSE also if the hostname is
a fully qualified domain name (FQDN), that is, it "contains
periods". For instance, c('node1', 'node2.server.org') will use
homogeneous = TRUE for the first worker and homogeneous = FALSE
for the second.
makeClusterPSOCK() now asserts that each cluster node is
functioning by retrieving and recording the node's session
information including the process ID of the corresponding R
process.
Nested futures sets option mc.cores to prevent spawning of
recursive parallel processes by mistake. Because 'mc.cores'
controls additional processes, it was previously set to zero.
However, since some functions
such as mclapply() does not support that, it is now set to one instead.
makeClusterPSOCK() gained more detailed descriptions on
arguments and what their defaults are.future_lapply() with multicore / multisession futures, would use
a suboptimal workload balancing where it split up the data in one
chunk too many. This is no longer a problem because of how
argument workers is now defined for those type of futures (see
note on top).
future_lapply(), as well as lazy multicore and lazy sequential
futures, did not respect option future.globals.resolve, but was
hardcoded to always resolve globals (future.globals.resolve = TRUE).
When globals larger than the allowed size (option
future.globals.maxSize) are detected an informative error message
is generated. Previous version introduced a bug causing the error
to produce another error.
Lazy sequential futures would produce an error when resolved if required packages had been detached.
print() would not display globals gathered for lazy sequential
futures.
Added package tests for globals part of formulas part of other
globals, e.g. purrr::map(x, ~ rnorm(.)), which requires
globals (>= 0.10.0).
Now package tests with parallel::makeCluster() not only test for
type = "PSOCK" clusters but also "FORK" (when supported).
TESTS: Cleaned up test scripts such that the overall processing time for the tests was roughly halved, while preserving the same test coverage.
future_lapply() is now to not generate RNG
seeds (future.seed = FALSE). If proper random number generation
is needed, use future.seed = TRUE. For more details, see help
page.future() and future_lapply() gained argument packages for
explicitly specifying packages to be attached when the futures are
evaluated. Note that the default throughout the future package
is that all globals and all required packages are automatically
identified and gathered, so in most cases those do not have to be
specified manually.
The default values for arguments connectTimeout and timeout of
makeNodePSOCK() can now be controlled via global options.
Now future_lapply() guarantees that the RNG state of the calling
R process after returning is updated compared to what it was before
and in the exact same way regardless of future.seed (except
FALSE), future.scheduling and future strategy used. This is done
in order to guarantee that an R script calling future_lapply()
multiple times should be numerically reproducible given the same
initial seed.
It is now possible to specify a pre-generated sequence of
.Random.seed seeds to be used for each FUN(x[[i]], ...) call in
future_lapply(x, FUN, ...).
future_lapply() scans global variables for non-resolved futures
(to resolve them) and calculate their total size once. Previously,
each chunk (a future) would redo this.Now future_lapply(X, FUN, ...) identifies global objects among
X, FUN and ... recursively until no new globals are found.
Previously, only the first level of globals were scanned. This is
mostly thanks to a bug fix in globals 0.9.0.
A future that used a global object x of a class that overrides
length() would produce an error if length(x) reports more
elements than what can be subsetted.
nbrOfWorkers() gave an error with plan(cluster, workers = cl)
where cl is a cluster object created by
parallel::makeCluster(), etc. This prevented for instance
future_lapply() to work with such setups.
plan(cluster, workers = cl) where cl <- makeCluster(..., type = MPI") would give an instant error due to an invalid internal
assertion.
Previously deprecated arguments maxCores and cluster are now
defunct.
Previously deprecated assignment operators %<=% and %=>% are
now defunct.
availableCores(method = "mc.cores") is now defunct in favor of
"mc.cores+1".
plan(),
e.g. plan(cluster) will set up workers on all cluster nodes.
Previously, this only happened when the first future was created.Renamed 'eager' futures to 'sequential', e.g. plan(sequential).
The 'eager' futures will be deprecated in an upcoming release.
Added support for controlling whether a future is resolved eagerly
or lazily when creating the future, e.g. future(..., lazy = TRUE), futureAssign(..., lazy = TRUE), and x %<-% { ... } %lazy% TRUE.
future(), futureAssign() and futureCall() gained argument
seed, which specifies a L'Ecuyer-CMRG random seed to be used by
the future. The seed for future assignment can be specified via
%seed%.
futureAssign() now passes all additional arguments to future().
Added future_lapply() which supports load balancing ("chunking")
and perfect reproducibility (regardless of type of load balancing
and how futures are resolved) via initial random seed.
Added availableWorkers(). By default it returns localhost
workers according to availableCores(). In addition, it detects
common HPC allocations given in environment variables set by the
HPC scheduler.
The default for plan(cluster) is now workers = availableWorkers().
Now plan() stops any clusters that were implicitly created. For
instance, a multisession cluster created by plan(multisession)
will be stopped when plan(eager) is called.
makeClusterPSOCK() treats workers that refer to a local machine
by its local or canonical hostname as "localhost". This avoids
having to launch such workers over SSH, which may not be supported
on all systems / compute cluster.
Option future.debug = TRUE also reports on total size of globals
identified and for cluster futures also the size of the individual
global variables exported.
Option future.wait.timeout (replaces future.wait.times)
specifies the maximum waiting time for a free workers (e.g. a core
or a
compute node) before generating a timeout error.
Option future.availableCores.fallback, which defaults to
environment variable R_FUTURE_AVAILABLECORES_FALLBACK can now be
used to specify the default number of cores / workers returned by
availableCores() and availableWorkers() when no other settings
are available. For instance, if
R_FUTURE_AVAILABLECORES_FALLBACK=1 is set system wide in an HPC
environment, then all R processes that uses availableCores() to
detect how many cores can be used will run as single-core
processes. Without this fallback setting, and without other
core-specifying settings, the default will be to use all cores on
the machine, which does not play well on multi-user systems.
plan(lazy) are now deprecated. Instead, use
plan(eager) and then f <- future(..., lazy = TRUE) or x %<-% { ... } %lazy% TRUE. The reason behind this is that in some cases
code that uses futures only works under eager evaluation (lazy = FALSE; the default), or vice verse. By removing the "lazy" future
strategy, the user can no longer
override the lazy = TRUE / FALSE that the developer is using.Creation of cluster futures (including multisession ones) would
time out already after 40 seconds if all workers were busy. New
default timeout is 30 days (option future.wait.timeout).
nbrOfWorkers() gave an error for plan(cluster, workers) where
workers was a character vector or a cluster object of the
parallel package. Because of this, future_lapply() gave an
error with such setups.
availableCores(methods = "_R_CHECK_LIMIT_CORES_") would give an
error if not running R CMD check.
Added makeClusterPSOCK() - a version of
parallel::makePSOCKcluster() that allows for more flexible
control of how PSOCK cluster workers are set up and how they are
launched and communicated with if running on external machines.
Added generic as.cluster() for coercing objects to cluster
objects to be used as in plan(cluster, workers = as.cluster(x)).
Also added a c() implementation for cluster objects such that
multiple cluster objects can be combined into a single one.
Added sessionDetails() for gathering details of the current R
session.
plan() and plan("list") now prints more user-friendly output.
On Unix, internal myInternalIP() tries more alternatives for
finding the local IP number.
%<=% is deprecated. Use %<-% instead. Same for %=>%.values() for lists and list environments of futures where one or
more of the futures resolved to NULL would give an error.
value() for ClusterFuture would give cryptic error message
"Error in stop(ex) : bad error message" if the cluster worker had
crashed / terminated. Now it will instead give an error message
like "Failed to retrieve the value of ClusterFuture from cluster
node #1 on 'localhost'. The reason reported was "error reading from
connection".
Argument user to remote() was ignored (since 1.1.0).
workers = "localhost" they (again) use the exact same R executable as the
main / calling R session (in all other cases it uses whatever
Rscript is found in the PATH). This was already indeed
implemented in 1.0.1, but with the added support for reverse SSH
tunnels in 1.1.0 this default behavior was lost.REMOTE CLUSTERS: It is now very simple to use cluster() and
remote() to connect to remote clusters / machines. As long as
you can connect via SSH to those machines, it works also with these
future. The new code completely avoids incoming firewall and
incoming port forwarding issues previously needed. This is done by
using reverse SSH tunneling. There is also no need to worry about
internal or external IP numbers.
Added optional argument label to all futures, e.g. f <- future(42, label = "answer") and v %<-% { 42 } %label% "answer".
Added argument user to cluster() and remote().
Now all Future classes supports run() for launching the future
and value() calls run() if the future has not been launched.
MEMORY: Now plan(cluster, gc = TRUE) causes the background R
session to be garbage collected immediately after the value is
collected. Since multisession and remote futures are special cases
of cluster futures, the same is true for these as well.
ROBUSTNESS: Now the default future strategy is explicitly set when
no strategies are set, e.g. when used nested futures. Previously,
only mc.cores was set so that only a single core was used, but now
also plan("default") set.
WORKAROUND: resolved() on cluster futures would block on Linux
until future was resolved. This is due to a bug in R. The
workaround is to use round the timeout (in seconds) to an integer,
which seems to always work / be respected.
Global variables part of subassignments in future expressions are
recognized and exported (iff found), e.g. x$a <- value, x[["a"]] <- value, and x[1,2,3] <- value.
Global variables part of formulae in future expressions are
recognized and exported (iff found), e.g. y ~ x | z.
As an alternative to the default automatic identification of
globals, it is now also possible to explicitly specify them either
by their names (as a character vector) or by their names and values
(as a named list), e.g. f <- future({ 2*a }, globals = c("a")) or
f <- future({ 2*a }, globals = list(a = 42)). For future
assignments one can use the %globals% operator, e.g. y %<-% { 2*a } %globals% c("a").
ROBUSTNESS: For the special case where 'remote' futures use
workers = "localhost" they now use the exact same R executable as
the main / calling R session (in all other cases it uses whatever
Rscript is found in the PATH).
FutureError now extends simpleError and no longer the error
class of captured errors.
Since future 0.13.0, a global pkg would be overwritten by the
name of the last package attached in future.
Futures that generated R.oo::Exception errors, they triggered
another internal error.
Add support for remote(..., myip = "<external>"), which now
queries a set of external lookup services in case one of them
fails.
Add mandelbrot() function used in demo to the API for
convenience.
ROBUSTNESS: If .future.R script, which is sourced when the
future package is attached, gives an error, then the error is
ignored with a warning.
TROUBLESHOOTING: If the future requires attachment of packages,
then each namespace is loaded separately and before attaching the
package. This is done in order to see the actual error message in
case there is a problem while loading the namespace. With
require()/library() this error message is otherwise suppressed
and replaced with a generic one.
Falsely identified global variables no longer generate an error
when the future is created. Instead, we leave it to R and the
evaluation of the individual futures to throw an error if the a
global variable is truly missing. This was done in order to
automatically handle future expressions that use non-standard
evaluation (NSE), e.g. subset(df, x < 3) where x is falsely
identified as a global variable.
Dropped support for system environment variable
R_FUTURE_GLOBALS_MAXSIZE.
DEMO: Now the Mandelbrot demo tiles a single Mandelbrot region with one future per tile. This better illustrates parallelism.
Documented R options used by the future package.
Custom futures based on a constructor function that is defined outside a package gave an error.
plan("default") assumed that the future.plan option was a
string; gave an error if it was a function.
Various future options were not passed on to futures.
A startup .future.R script is no longer sourced if the
future package is attached by a future expression.
Added remote futures, which are cluster futures with convenient
default arguments for simple remote access to R, e.g.
plan(remote, workers = "login.my-server.org").
Now .future.R (if found in the current directory or otherwise in
the user's home directory) is sourced when the future package
is attach (but not loaded). This helps separating scripts from
configuration of futures.
Added support for plan(cluster, workers = c("n1", "n2", "n2", "n4")), where workers (also for ClusterFuture()) is a set of
host names passed to parallel::makeCluster(workers). It can also
be the number of localhost workers.
Added command line option --parallel=<p>, which is long for -p <p>.
Now command line option -p <p> also set the default future
strategy to multiprocessing (if p >= 2 and eager otherwise), unless
another strategy is already specified via option future.plan or
system environment variable R_FUTURE_PLAN.
Now availableCores() also acknowledges environment variable
NSLOTS set by Sun/Oracle Grid Engine (SGE).
MEMORY: Added argument gc = FALSE to all futures. When TRUE, the
garbage collector will run at the very end in the process that
evaluated the future (just before returning the value). This may
help lowering the overall memory footprint when running multiple
parallel R processes. The user can enable this by specifying
plan(multiprocess, gc = TRUE). The developer can control this
using future(expr, gc = TRUE) or v %<-% { expr } %tweak% list(gc = TRUE).
plan(list("eager")), whereas it
did work with plan("eager") and plan(list(eager)).Added nbrOfWorkers().
Added informative print() method for the Future class.
values() passes arguments ... to value() of each future.
Added FutureError class.
maxCores and cluster to workers. If using
the old argument names a deprecation warning will be generated, but
it will still work until made defunct in a future release.resolve() for lists and environments did not work
properly when the set of futures was not resolved in order,
which could happen with asynchronous futures.Add support to plan() for specifying different future strategies
for the different levels of nested futures.
Add backtrace() for listing the trace the expressions evaluated
(the calls made) before a condition was caught.
Add transparent futures, which are eager futures with early signaling of conditioned enabled and whose expression is evaluated in the calling environment. This makes the evaluation of such futures as similar as possible to how R evaluates expressions, which in turn simplifies troubleshooting errors, etc.
Add support for early signaling of conditions. The default is (as before) to signal conditions when the value is queried. In addition, they may be signals as soon as possible, e.g. when checking whether a future is resolved or not.
Signaling of conditions when calling value() is now controlled by
argument signal (previously onError).
Now UniprocessFuture:s captures the call stack for errors
occurring while resolving futures.
ClusterFuture() gained argument persistent = FALSE. With
persistent = TRUE, any objects in the cluster R session that was
created during the evaluation of a previous future is available for
succeeding futures that are evaluated in the same session.
Moreover, globals are still identified and exported but "missing"
globals will not give an error - instead it is assumed such globals
are available in the environment where the future is evaluated.
OVERHEAD: Utility functions exported by ClusterFuture are now
much smaller; previously they would export all of the package
environment.
f <- multicore(NA, maxCores = 2) would end up in an endless
waiting loop for a free core if availableCores() returned one.
ClusterFuture() would ignore local = TRUE.
Added multiprocess futures, which are multicore futures if
supported, otherwise multisession futures. This makes it possible
to use plan(multiprocess) everywhere regardless of operating
system.
Future strategy functions gained class attributes such that it is
possible to test what type of future is currently used, e.g.
inherits(plan(), "multicore").
ROBUSTNESS: It is only the R process that created a future that can resolve it. If a non-resolved future is queried by another R process, then an informative error is generated explaining that this is not possible.
ROBUSTNESS: Now value() for multicore futures detects if the
underlying forked R process was terminated before completing and if
so generates an informative error messages.
resolve() gained argument recursive.
Added option future.globals.resolve for controlling whether
global variables should be resolved for futures or not. If TRUE,
then globals are searched recursively for any futures and if found
such "global" futures are resolved. If FALSE, global futures are
not located, but if they are later trying to be resolved by the
parent future, then an informative error message is generated
clarifying that only the R process that created the future can
resolve it. The default is currently FALSE.
FIX: Exports of objects available in packages already attached by the future were still exported.
FIX: Now availableCores() returns 3L (=2L+1L) instead of 2L
if _R_CHECK_LIMIT_CORES_ is set.
Add multisession futures, which analogously to multicore ones, use multiple cores on the local machine with the difference that they are evaluated in separate R session running in the background rather than separate forked R processes. A multisession future is a special type of cluster futures that do not require explicit setup of cluster nodes.
Add support for cluster futures, which can make use of a cluster of
nodes created by parallel::makeCluster().
Add futureCall(), which is for futures what do.call() is
otherwise.
Standardized how options are named, i.e. future.<option>. If you
used any future options previously, make sure to check they follow
the above format.
globals = TRUE).Now %<=% can also assign to multi-dimensional list environments.
Add futures(), values() and resolved().
Add resolve() to resolve futures in lists and environments.
Now availableCores() also acknowledges the number of CPUs
allotted by Slurm.
CLEANUP: Now the internal future variable created by %<=% is
removed when the future variable is resolved.
futureOf(envir = x) did not work properly when x was a list
environment.ROBUSTNESS: Now values of environment variables are trimmed before being parsed.
ROBUSTNESS: Add reproducibility test for random number generation using Pierre L'Ecuyer's RNG stream regardless of how futures are evaluated, e.g. eager, lazy and multicore.
findGlobals(..., method = "ordered") in globals (> 0.5.0)
such that a global variable preceding a local variable with the
same name is properly identified and exported/frozen.Globals that were copies of package objects were not exported to the future environments.
The future package had to be attached or future::future() had
to be imported, if %<=% was used internally in another package.
Similarly, it also had to be attached if multicore futures where
used.
eager() and multicore() gained argument globals, where
globals = TRUE will validate that all global variables identified
can be located already before the future is created. This provides
the means for providing the same tests on global variables with
eager and multicore futures as with lazy futures.lazy(sum(x, ...), globals = TRUE) now properly passes ... from
the function from which the future is setup. If not called within
a function or called within a function without ... arguments, an
informative error message is thrown.plan("default") resets to the default strategy, which is
synchronous eager evaluation unless option future_plan or
environment variable R_FUTURE_PLAN has been set.
availableCores("mc.cores") returns getOption("mc.cores") + 1L,
because option mc.cores specifies "allowed number of additional
R processes" to be used in addition to the main R process.
plan(future::lazy) and similar gave errors.multicore() gained argument maxCores, which makes it possible
to use for instance plan(multicore, maxCores = 4L).
Add availableMulticore() [from (in-house) async package].
demo("mandelbrot", package = "future").ROBUSTNESS: multicore() blocks until one of the CPU cores is
available, iff all are currently occupied by other multicore
futures.
old <- plan(new) now returns the old plan/strategy (was the newly
set one).
Eager and lazy futures now records the result internally such that the expression is only evaluated once, even if their error values are requested multiple times.
Eager futures are always created regardless of error or not.
All Future objects are environments themselves that record the
expression, the call environment and optional variables.
lazy() "freezes" global variables at the time when the future is
created. This way the result of a lazy future is more likely to be
the same as an 'eager' future. This is also how globals are likely
to be handled by asynchronous futures.plan() records the call.demo("mandelbrot", package = "future"), which can be
re-used by other future packages.Added plan().
Added eager future - useful for troubleshooting.