A Future for batchtools

Introduction

The future package provides a generic API for using futures in R. A future is a simple yet powerful mechanism to evaluate an R expression and retrieve its value at some point in time. Futures can be resolved in many different ways depending on which strategy is used. There are various types of synchronous and asynchronous futures to choose from in the future package.

This package, future.batchtools, provides a type of futures that utilizes the batchtools package. This means that any type of backend that the batchtools package supports can be used as a future. More specifically, future.batchtools will allow you or users of your package to leverage the compute power of high-performance computing (HPC) clusters via a simple switch in settings - without having to change any code at all.

For instance, if batchtools is properly configures, the below two expressions for futures x and y will be processed on two different compute nodes:

> library("future.batchtools")
> plan(batchtools_torque)
>
> x %<-% { Sys.sleep(5); 3.14 }
> y %<-% { Sys.sleep(5); 2.71 }
> x + y
[1] 5.85

This is just a toy example to illustrate what futures look like and how to work with them.

A more realistic example comes from the field of cancer research where very large data FASTQ files, which hold a large number of short DNA sequence reads, are produced. The first step toward a biological interpretation of these data is to align the reads in each sample (one FASTQ file) toward the human genome. In order to speed this up, we can have each file be processed by a separate compute node and each node we can use 24 parallel processes such that each process aligns a separate chromosome. Here is an outline of how this nested parallelism could be implemented using futures.

library("future")
library("listenv")
## The first level of futures should be submitted to the
## cluster using batchtools.  The second level of futures
## should be using multisession, where the number of
## parallel processes is automatically decided based on
## what the cluster grants to each compute node.
plan(list(batchtools_torque, multisession))

## Find all samples (one FASTQ file per sample)
fqs <- dir(pattern = "[.]fastq$")

## The aligned results are stored in BAM files
bams <- listenv()

## For all samples (FASTQ files) ...
for (ss in seq_along(fqs)) {
  fq <- fqs[ss]

  ## ... use futures to align them ...
  bams[[ss]] %<-% {
    bams_ss <- listenv()
    ## ... and for each FASTQ file use a second layer
    ## of futures to align the individual chromosomes
    for (cc in 1:24) {
      bams_ss[[cc]] %<-% htseq::align(fq, chr = cc)
    }
    ## Resolve the "chromosome" futures and return as a list
    as.list(bams_ss)
  }
}
## Resolve the "sample" futures and return as a list
bams <- as.list(bams)

Note that a user who do not have access to a cluster could use the same script processing samples sequentially and chromosomes in parallel on a single machine using:

plan(list(sequential, multisession))

or samples in parallel and chromosomes sequentially using:

plan(list(multisession, sequential))

For an introduction as well as full details on how to use futures, please consult the package vignettes of the future package.

Choosing batchtools backend

The future.batchtools package implements a generic future wrapper for all batchtools backends. Below are the most common types of batchtools backends.

#!/bin/bash

## Job name:
#PBS -N <%= if (exists("job.name", mode = "character")) job.name else job.hash %>

## Direct streams to logfile:
#PBS -o <%= log.file %>

## Merge standard error and output:
#PBS -j oe

## Email on abort (a) and termination (e), but not when starting (b)
#PBS -m ae

## Resources needed:
<% if (length(resources) > 0) {
  opts <- unlist(resources, use.names = TRUE)
  opts <- sprintf("%s=%s", names(opts), opts)
  opts <- paste(opts, collapse = ",") %>
#PBS -l <%= opts %>
<% } %>

## Launch R and evaluated the batchtools R job
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'

> plan(batchtools_torque)

> plan(batchtools_torque, resources = list(nodes = "1:ppn=12", vmem = "5gb"))

plan(list(
  tweak(batchtools_torque, resources = list(nodes = "1:ppn=12", vmem = "5gb")),
  multisession
))

> plan(batchtools_slurm)

> plan(batchtools_slurm, template = "/path/to/batchtools.slurm.tmpl")

Demos

The future package provides a demo using futures for calculating a set of Mandelbrot planes. The demo does not assume anything about what type of futures are used. The user has full control of how futures are evaluated. For instance, to use local batchtools futures, run the demo as:

library("future.batchtools")
plan(batchtools_local)
demo("mandelbrot", package = "future", ask = FALSE)