src.rmd

---
layout: default
title: Compiled code
output: bookdown::html_chapter
---

# Compiled code {#src}

It's often useful to include compiled code in an R package. Compiled code, usually  C or C++, is a powerful complement to R code. R code is high-level and expressive, but that expressivity comes at a cost: speed. Low-level compiled languages like C and C++ often require more typing (and more thinking) to solve a problem, they can be orders of magnitude faster than R. 

Unfortunately, teaching you how to program in C or C++ is beyond the scope of the book. If you'd like to learn, I recommmend starting with C++ and the Rcpp package. Rcpp makes it very easy to connect C++ to R, and RStudio has many tools to facilitate the process. Start by reading my ["High performance functions with Rcpp"](http://adv-r.had.co.nz/Rcpp.html), a freely available book chapter from [Advanced R](http://amzn.com/1466586966?tag=devtools-20): it gently introduces you to C++ by translating familiar R code. Next, check out the [Rcpp book](http://www.rcpp.org/book) and the other resources listed in [learning more](http://adv-r.had.co.nz/Rcpp.html#rcpp-more).

## C++ {#cpp}

To set up your package with Rcpp, run:

```{r, eval = FALSE}
devtools::use_rcpp()
```

This will:

* Create a `src/` directory to hold your `.cpp` files.

* Add `Rcpp` to the `LinkingTo` and `Imports` fields in the `DESCRIPTION`.

* Set up a `.gitignore` file to make sure you don't accidentally check in
  any compiled files (learn more about this in [git](#git)).

* Tell you the two roxygen tags you need to add to your package:

    ```{r}
    #' @useDynLib your-package-name
    #' @importFrom Rcpp sourceCpp
    NULL
    ```

### Workflow {#cpp-workflow}

Once you're set up, the basic workflow is familiar:

1.  Create a new C++ file:

    ```{r, echo = FALSE}
    bookdown::embed_png("screenshots/new-cpp.png", dpi = 220)
    ```

    The default template looks like this:
    
    ```cpp
    #include <Rcpp.h>
    using namespace Rcpp;
    
    // Below is a simple example of exporting a C++ function to R. You can
    // source this function into an R session using the Rcpp::sourceCpp 
    // function (or via the Source button on the editor toolbar)
    
    // For more on using Rcpp click the Help button on the editor toolbar
    
    // [[Rcpp::export]]
    int timesTwo(int x) {
       return x * 2;
    }
    ```
    
    It includes a basic function and some instructions to get started. The
    two most important parts are the header includes, and the special
    attribute `// [[Rcpp::export]]`.

1.  Generate the necessary modifications to your `NAMESPACE` by documenting
    with Cmd + Shift + D.

1.  Press Cmd + Shift + L to load all code (or Cmd + Shift + B to
    build and reload).

1.  Run `timesTwo(10)` from the console to check that it works.

Behind the scenes, `devtools::load_all()` and RStudio's "Build and reload" do a lot of work for you. They:

* Set up your R environment to compile code and warn you if you're missing 
  necessary pieces.
  
* Call `Rcpp::compileAttributes()`. This inspects your `.cpp` functions 
  looking for __attributes__ of the form `// [[Rcpp::export]]`. When it finds 
  one, it generates the code necessary to make the function available in R, 
  and creates `src/RcppExports.cpp` and `R/RcppExports.R`. You should never 
  modify these files by hand.
  
* Build a DLL (dynamically linked library) and make it available to R.

### Documentation {#cpp-man}

Each C++ function that is exported has an automatically written wrapper function that lives in `R/RcppExports.R`. For example, the R `timesTwo()` function looks like:

```{r}
timesTwo <- function(x) {
  .Call('timesTwo', PACKAGE = 'mypackage', x)
}
```

This uses the base function `.Call()` to execute the C function `timesTwo` provided by "mypackage". You can use roxygen2 to document this like a regular R function, but instead of using `#'` you use `//'` (the C++ commenting convention):

```cpp
//' Multiply a number by two
//' 
//' @param x A single integer.
//' @export
// [[Rcpp::export]]
int timesTwo(int x) {
   return x * 2;
}
```

That generates roxygen comments in `R/RcppExports.R`:

```{r}
#' Multiply a number by two
#' 
#' @param x A single integer.
#' @export
timesTwo <- function(x) {
  .Call('timesTwo', PACKAGE = 'mypackage', x)
}
```

The distinctions between the two export directives is important:

* `[[Rcpp::export]]`: makes the C++ function available to R. If you have
  trouble rememebering the exact details, note that everything comes in 
  twos: Two `\`, two `[`, two `:` and two `]`.

* `@export`: makes the R wrapper function available outside your package,
  adding it to the `NAMESPACE`.

### Exporting C++ code {#cpp-export}

To make your C++ code callable from C++ code in other packages, add:

```cpp
// [[Rcpp::interfaces(r, cpp)]]
```

This will generate a header file, `inst/include/mypackage.h` that can be included by other packages (The low-level details are described in [Exporting C code]{#c-export}). See "[Rcpp Attributes](http://dirk.eddelbuettel.com/code/rcpp/Rcpp-attributes.pdf)" for more details, including how to combine hand-written and automatically generated header files.

### Importing C++ code {#cpp-import}

To use C++ code from another package:

1. In `DESCRIPTION`, add `LinkingTo: otherPackage`. Confusingly this has nothing
   to do with linking, but instead adds `otherPackage/include` to the include 
   path.

1. In the C++ file, add:

    ```cpp
    #include <otherPackage.h>
    ```

1. Functions from otherPackage will be included in the `otherPackage` C++
   namespace. Use `otherPackage::foo()` to access functions, or make 
   them available globally with `using namespace otherPackage`.

### Best practices {#cpp-best-practices}

* To print output use `Rcout << ...` (not `cout << ...`). This prints to 
  the right place, which might be a GUI console or a file (if `sink()` 
  is active)

* In long-running loops, regularly run `Rcpp::checkUserInterrupt()`. This 
  aborts your C++ if the user has pressed Ctrl + C or Escape in R.

* Use `.h` extension for headers and include files. (If you don't
  `R CMD check` will complain).
  
* Follow Martyn Plummer's recommendations on 
  [Portable C++ for R packages](http://journal.r-project.org/archive/2011-2/RJournal_2011-2_Plummer.pdf).

* Whenever you use C++ code in your package, you need to clean up after 
  yourself when your package is unloaded. Do this by writing a `.onUnload()` 
  function that unloads the DLL:

    ```{r}
    .onUnload <- function (libpath) {
      library.dynam.unload("mypackage", libpath)
    }
    ```

* Use `clang` instead of `gcc` to compile your C++ code: it gives much 
  better error messages. You can make `clang` the default by creating a
  `~/.R/Makevars` that contains:

    ```bash
    CXX=clang++
    ```

* To speed up compilation, install `ccache`, then replace `~/.R/Makevars`
  with:

    ```bash
    CC=ccache clang -Qunused-arguments
    CXX=ccache clang++ -Qunused-arguments
    CCACHE_CPP2=yes
    ```

## C

If you're writing new compiled code, it's almost always better to use Rcpp. It's less work, more consistent, better documented, and has better tooling. However, there are some reasons you might choose to use C:

* You're working with an older package that already uses the C API.
* You're binding to an existing C library.

### Getting started

To call a C function from R, you first need a C function! In R packages, C code lives in `.c` files in `src/`. It needs to include two header files:

```c
#include <R.h>
#include <Rinternals.h>
```

(Including `<Rinternals.h>` seems like bad form, but it doesn't actually give you access to the "internal" internal API unless you set some additional flags. The default just gives you the "public" internal API which is both safe and necessary. Yes, this is confusing.)

These headers allow you to access R's C API. Unfortunately this API is not well documented. I'd recommend starting with my notes at [R's C interface](http://adv-r.had.co.nz/C-interface.html). After that, read "[The R API](http://cran.rstudio.com/doc/manuals/r-devel/R-exts.html#The-R-API)" in "Writing R Extensions". A number of exported functions are not documented, so you'll also need to read the [R source code](https://github.com/wch/r-source) to figure out the details.

Here's the minimum you need to know: C functions that talk to R must use the `SEXP` type for both inputs and outputs. `SEXP`, short for S expression, is the C struct used to represent every type of object in R. A C function typically starts by converting `SEXP`s to C objects, and ends by converting C objects back to a `SEXP`. (The R API is designed so that these conversions often don't require copying.) The following table lists the functions that convert length one R vectors to and from C scalars:

R type      | C type      | R -> C            |  C -> R 
------------|-------------|-------------------|-------------------
integer     | int         | `asInteger(x)`    | `ScalarInteger(x)`
numeric     | double      | `asReal(x)`       | `ScalarReal(x)`
logical     | int         | `asLogical(x)`    | `ScalarLogical(x)`
chararacter | const char* | `CHAR(asChar(x))` | `mkString(x)`

We now have enough information to write a simple C function that can add two numbers together:

```c
#include <R.h>
#include <Rinternals.h>

SEXP add_(SEXP x_, SEXP y_) {
  double x = asReal(x_);
  double y = asReal(y_);
  
  double sum = x + y;
  
  return ScalarReal(sum);
}
```

To call a C function from R, use `.Call()`:

```{r}
#' @useDynLib mypackage add_
add <- function(x, y) .Call(add_, x, y)
```

There are two ways to call C functions from R: `.C()` and `.Call()`. `.C()` is an older interface and should not be used for new code. If you want to learn about it, you'll need to read [Interface .C and .Fortran functions](http://cran.r-project.org/doc/manuals/R-exts.html#Interface-functions-_002eC-and-_002eFortran) in "Writing R extensions". All modern code should use `.Call()`. 

Where does the first argument to `.Call()`, `add_`, come from? This is thanks to `@useDynLib` which creates a line in the NAMESPACE that looks like:

```
useDynLib(mypackage, add_)
```

This directive instructs R to create an object called `add_` which describes a C function pointer:

```{r, eval = FALSE}
add_
#> $name
#> [1] "add_"
#> 
#> $address
#> <pointer: 0x107be3f40>
#> $package
#> NULL
#> 
#> attr(,"class")
#> [1] "NativeSymbolInfo"
```

`.Call()` takes pointer to a C function and calls it. It does not check the arguments, so you need to do that in the R function, in the C function, or just accept that R will crash every time you accidentally supply the wrong type of input.

### Workflow {#c-workflow}

The usual workflow still applies:

1.  Modify the C code.
1.  Load all code with Cmd + Shift + L or Build and Reload with Cmd + Shift + B
1.  Experiment at the console.

The first time you add `@useDynLib`, you'll also need to run `devtools::document()` (Cmd + Shift + D) and reload the package.

### Exporting C code {#c-export}

R packages need to provide relocatable DLLs; DLLs that work regardless of where they live on disk. This is because most R users don't build packages from source. Instead, they get binaries from CRAN, and they are installed in many different places. The need for relocatable DLLs makes importing and export C code for R packages require a few more steps (the same problem arises for C++, but Rcpp attributes automate the manual steps I describe below). 

R solves this problem using __function registration__. To export a C function, you register it with `R_RegisterCCallable()`. To import a C function, you get a pointer to it with `R_GetCCallable()`. As we'll see below, a user-friendly package will do both these things, so users of the package can ignore the details and simply include a header a file.

[Sidebar: Confusingly, there's another type of function registration. Instead of registering C functions using the namespace (i.e. `@useDynLib pkg fun`), you can register them with `R_registerRoutines()` and `@useDynLib mypackage, .registration = TRUE`. This is only useful if useful if you're using the older `.C` interface, or the more esoteric `.Fortran` and `.External` interfaces. To learn the details read [Registering native extensions](http://cran.r-project.org/doc/manuals/R-exts.html#Registering-native-routines) in "Writing R extensions".]

To register a function, call `R_RegisterCCallable()`, defined in `<R_ext/Rdynload.h>`. Function registration should be done in a function called `R_init_<mypackage>`. This function is called automatically when the "mypackage" DLL is loaded. `R_RegisterCCallable()` has three arguments:

* A pointer to the DLL.
* The name of the function.
* A pointer to the function, cast as `DL_FUNC` (i.e. a **d**ynamically 
  **l**oaded **func**tion).

The following code registers the `add()` function defined above:

```c
#include "add.h"
#include <R_ext/Rdynload.h>

void R_init_mypackage(DllInfo *info) {
  R_RegisterCCallable(info, "add_",  (DL_FUNC) &add_)
}
```

It doesn't matter where this code lives, but it's usually put in a file called `src/mypackage-init.c`. 

To access a registered function from another package, can `R_GetCCallable()`. It has two arguments, the function name and the package name, and it returns a function pointer. The function pointer has no type information, so it should always be wrapped in a helper function that defines the inputs:

```c
#include <R_ext/Rdynload.h>
#include <R.h>
#include <Rinternals.h>

SEXP add_(SEXP x, SEXP y) {
  static SEXP(fun*)(SEXP, SEXP) = NULL;
  if (fun == NULL)
    fun = (SEXP(*)(SEXP, SEXP)) R_GetCCallable("add", "mypackage");
  return fun(x, y);
}
```

Rather than relying on each package that imports your C code to do this correctly, you should do it for them. Write `inst/include/mypackageAPI.h` which provides a wrapper function for each exported function. A popular package that does that is [xts](http://cran.r-project.org/web/packages/xts/). Download the source package and look in the `include/` directory to see what it does.

### Importing C code {#c-import}

Using C code from another package varies based on how the package is implemented:

* If it uses the system descibed above, all you need is`LinkingTo: otherPackage` 
  in the `DESCRIPTION`, and `#include mypackageAPI.h` in the C file. 
  (Remember `LinkingTo` is not about linking, but actually affects the include 
  path).
   
* If it registers the functions, but doesn't provide a header file, you'll
  need to write the wrapper yourself. Since you're not using any header
  files from the package, you use `Imports` and not `LinkingTo`. You 
  also need to make sure the package is loaded. You can do this by importing
  any function with `@importFrom mypackage foo`, or by adding 
  `requireNamespace("mypackage", quietly = TRUE)` to `.onLoad()`.
  
* If it doesn't register the functions, you can't use them. You'll have to
  ask the maintainer nicely or even provide a pull request.

### Best practices {#c-best-practices}

* Avoid calls to `assert()`, `abort()` and `exit()`: these will kill the 
  R process, not just your C code. Instead, use `error()` which is 
  equivalent to calling `stop()` in R.

* To print output use `Rprintf()`, not `printf()`. This always prints to 
  the right place, which might be a GUI console, or a file (if `sink()` 
  is active).

* In long-running loops, regularly call `R_CheckUserInterrupt()` to allow
  the user to interrupt C code.

* Don't use C's random number generators (like `rand()` or `random()`), 
  instead use the C API to R's rngs: `unif_rand()`, `norm_rand()`, etc. 
  Note the caveats in ["Random number generation"](http://cran.rstudio.com/doc/manuals/r-devel/R-exts.html#Random-numbers) - you must call `GetRNGstate()` before and 
  `PutRNGstate()` after.

* Use R macros `ISNAN(x)` and `R_FINITE(x)` to check for NaNs and infinite
  values. These work on more platforms than the C99 `isnan()` and `isfinite()`.

* Like with C++, whenever you use C code in your package, you should unload the 
  DLL when the package is unloaded:

    ```{r}
    .onUnload <- function (libpath) {
      library.dynam.unload("mypackage", libpath)
    }
    ```

* Use `clang` instead of `gcc` to compile your C code: it gives much 
  better error messages. You can make `clang` the default by creating a
  `~/.R/Makevars` that contains:

    ```bash
    C=clang
    ```

## Debugging compiled code {#src-debugging}

It's possible, with a little extra work, to use an interactive debugger to debug your C/C++ in the same way that you can use `browser()` and `debug()` to debug your R code. Unfortunately you won't be able to use RStudio, and you'll have to run R from the command line. 

Open a shell (e.g. with Tools | Shell...) and start R by typing:

```bash
# If you compile with clang
R --debugger=lldb
# If you compile with gcc
R --debugger=gdb
```

This will start either [lldb](http://lldb.llvm.org) or [gdb](http://www.gnu.org/software/gdb/), the debuggers that work with code produced by `clang` or `gcc` respectively. Like R, `lldb` and `gdb` provide a REPL, a run-eval-print loop where you enter commands and then look at the results. In the examples below I'll show the results of `lldb` (because that's what I use), the output from `gdb` is similar. For each interactive command I'll tell you the explicit, but long, `lldb` command and the short, but cryptic, `gdb` command. `lldb` understand all `gdb` commands so you can use choose to be explicit of terse.

Once you've started the debugger, the first run you need to do is tell it to start R by typing `process start` or `run`. Now when your C/C++ code crashes you'll be dumped into an interactive debugger instead of getting a cryptic error message and a crash. 

Let's start with a simple C++ function designed to write to memory that doesn't belong to it:

```{r, eval = FALSE}
Rcpp::cppFunction("
bool mistake() {
  NumericVector x(1);
  int n = INT_MAX;
  x[n] = 0;
  return true;
}
", plugins = "debug", verbose = TRUE, rebuild = TRUE)
mistake()
```


Use `devtools::load_all()` to load the current package. Then copy and paste the code that creates the bug. Here's a crash from a package I was working on:

```
Process 32743 stopped
* thread #1: tid = 0x1f79f6, 0x0000000106a874e3 gggeom.so`vw_distance(x=0x00007fff5fbfcc50, y=0x00007fff5fbfcc40) + 499 at vw-distance.cpp:54, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x106a9cffc)
    frame #0: 0x0000000106a874e3 gggeom.so`vw_distance(x=0x00007fff5fbfcc50, y=0x00007fff5fbfcc40) + 499 at vw-distance.cpp:54
   51        int prev_idx = prev[idx];
   52  	
   53  	    next[prev[idx]] = next_idx;
-> 54  	    prev[next[idx]] = prev_idx;
   55  	    prev[idx] = -1;
   56  	    next[idx] = -1;
   57
```

It tells us that the crash because of a `EXC_BAD_ACCESS` - this is one of the most common types of crash in C/C++ code. Helpfully, lldb shows exactly which line of C++ code caused the problem: `vw-distance.cpp:54`. 

We're now at an interactive prompt can run the following commands:

* `help`: see a list of all commands. 

* `p cmd`: evaluate a C/C++ command in the current context

* `bt`

* `thread continue`, `thread step-in`, `thread step-over`, `thread step-out`

* `thread backtrace`

* `frame select`, `up`, `down`

* `frame variable`, `frame variable var`

(int) idx = <no location, value may have been optimized out>

Instead of waiting for a crash to occur you can also set breakpoints in your code.

Setting breakpoints

1. Press `Ctrl + C`

1. Type `breakpoint set --file foo.c --line 12 `

1. `c` to continue running.

This is also useful if your code is stuck in an infinite loop. Pretty `Ctrl + C` to break into the debugger and see which line of code is causing the problem.



## Makefiles {#make}

Makefiles are beyond the scope of this book, but they are a useful tool. A good, gentle introduction with with a reproducible research motivation is Karl Broman's ["Minimal make"](http://kbroman.org/minimal_make/).

Generally, R packages should avoid a custom `Makefile`. Instead, use `Makevars`. `Makevars` is a make file, but it's included before the R default make file generated by R (which is located at `file.path(R.home("etc"), "Makeconf")`). This allows you to take advantage of R's default behaviour (it's over 150 lines, and battle-tested across many years and many systems, so you want to!) while being able to set the flags you need. The most commonly used flags are:

* `PKG_LIBS`: Linker flags. A common use if `PKG_LIBS = $(BLAS_LIBS)`: that
  allows you to use the same BLAS library as R.

* `PKG_CFLAGS` & `PKG_CXXFLAGS`: C and C++ flags. Most commonly used to set 
  define directives with `-D`.

* `PKG_CPPFLAGS`: Pre-processor flags (not C++ flags!). Most commonly used to
  set include directories with `-I`. Any package listed in the `LinkingTo` field
  in the `DESCRIPTION` will be automatically included - you do not need to
  add it explicitly.

To set flags only on Windows, use `Makevars.win`. To build a `Makevars` with `configure`, use `Makevars.in`.

By default, R will use the system make, which is not always GNU compatible (i.e. on Solaris). If you want to use GNU extensions (which are extremely common), add `SystemRequirements: GNU make` to `DESCRIPTION`.  If you're not sure if you're using GNU extensions, play it safe add at the system requirement.

## Other languages {#src-other}

It is possible to connect R to other languages, although the interfaces are not as nice as for C++:

* __Fortran__: It's possible to call Fortran subroutines directly with 
  `.Fortran()`, or via C or C++ with `.Call()`. See `?.Fortran` and 

* __Java__: The [rJava](https://github.com/s-u/rJava) package makes it 
  possible to call Java code from within R. Note that unlike for C and C++,
  passing an R object to a java call must make a copy, which has serious
  performance implications.

## Licensing {#src-licensing}

When writing compiled code, it's common to use libraries written by other people. If you do so, must ensure that the package license is compatible with the license for the included code:

* The simplest solution is to use the same license for your code and the 
  included code. Since you can't relicense someone elses code, this means
  you need to change your license.

* If you don't want to use the same license, you're best sticking
  with common cases where the interactions are well known. For example, 
  [Various Licenses and Comments about Them](https://www.gnu.org/licenses/license-list.html)
  describes what licenses are compatible with the GPL license.
  
  In this case your description should contain 
  `License: <main license> + FILE license` where `<main license>` is a license 
  valid for the entire package (both R and compiled code), and the `license` 
  file describes the licenses of the individual components.

* For non-standard cases, you'll need to consult a lawyer.

In all cases, make sure you include copyright and license statements from the original code.

## CRAN issues {#src-cran}

Packages with compiled code are much more likely to have difficulties on CRAN than those without it. Your package must build from source on all major platforms (linux, windows and mac). This is hard! 

* CRAN provides an automated service for checking R packages on windows: 
  [win-builder](http://win-builder.r-project.org). You can access this easily
  by running `devtools::build_win()` which builds a package bundle and uploads
  it.
  
* I've tried to include the most important advice in this chapter, but I'd
  recommend reading the full section on [writing portable C and C++ code](http://cran.rstudio.com/doc/manuals/r-devel/R-exts.html#Portable-C-and-C_002b_002b-code) in "Writing
  R extensions"

* In exceptional circumstances, like binding to Windows only functionality, 
  you may be able to opt-out of cross-platform requirement, but expect that
  you'll have to make a strong case.

The interface between CRAN's automated and manual checking can be particularly frustrating for compiled code. Requirements vary from submission to submission, based on which maintainer you get and how much free time they have. The rules are inconsistently applied, but if they happen to you, it's best to bite the bullet and make the change rather than trying to argue about it:

* Sometimes you will need to list all authors and copyright holders of included 
  code in the DESCRIPTION. 

* Sometimes your package will need to work on Solaris. Fixing Solaris issues 
  can be hard, due to the difficulty of accessing a computer running Solaris. 
  You will be in a stronger negotiating position if the package has no problems 
  on other platforms. 
  
  One common gotcha: gcc/clang flags `-Wall`, `-pedantic` and `-O0` do not work 
  with the default compiler on Solaris.