---
title: "mirai - Torch Integration"
vignette: >
  %\VignetteIndexEntry{mirai - Torch Integration}
  %\VignetteEngine{litedown::vignette}
  %\VignetteEncoding{UTF-8}
---


### Torch Integration

Custom serialization functions may be registered to handle external pointer type reference objects.

This allows tensors from the [`torch`](https://torch.mlverse.org/) package to be used seamlessly in 'mirai' computations.

#### Setup Steps

1. Create the serialization configuration, specifying 'class' as 'torch_tensor' and 'vec' as TRUE.
1. Set up daemons, supplying the configuration to the 'serial' argument.
1. (Optional) Use `everywhere()` to make the `torch` package available on all daemons for convenience.


``` r
library(mirai)
library(torch)

cfg <- serial_config(
  class = "torch_tensor",
  sfunc = torch::torch_serialize,
  ufunc = torch::torch_load,
  vec = TRUE
)

daemons(1, serial = cfg)
#> [1] 1

everywhere(library(torch))
```
#### Example Usage

The below example creates a convolutional neural network using `torch::nn_module()`.

A set of model parameters is also specified.

The model specification and parameters are then passed to and initialized within a 'mirai'.


``` r
model <- nn_module(
  initialize = function(in_size, out_size) {
    self$conv1 <- nn_conv2d(in_size, out_size, 5)
    self$conv2 <- nn_conv2d(in_size, out_size, 5)
  },
  forward = function(x) {
    x <- self$conv1(x)
    x <- nnf_relu(x)
    x <- self$conv2(x)
    x <- nnf_relu(x)
    x
  }
)

params <- list(in_size = 1, out_size = 20)

m <- mirai(do.call(model, params), model = model, params = params)

m[]
#> An `nn_module` containing 1,040 parameters.
#> 
#> ── Modules ────────────────────────────────────────────────────────────────────────────────────────────────────────
#> • conv1: <nn_conv2d> #520 parameters
#> • conv2: <nn_conv2d> #520 parameters
```
The returned model is an object containing many tensor elements.

``` r
m$data$parameters$conv1.weight
#> torch_tensor
#> (1,1,.,.) = 
#>  -0.0744 -0.0110 -0.0537 -0.0237 -0.1781
#>  -0.1662 -0.0952 -0.1977 -0.1710 -0.0747
#>  -0.0216 -0.0314 -0.1124 -0.1277  0.0098
#>   0.1225  0.1188  0.1468  0.1675 -0.1111
#>   0.1517  0.1384  0.1313  0.0376 -0.0254
#> 
#> (2,1,.,.) = 
#>  -0.0459  0.0087 -0.0113 -0.0393 -0.1890
#>   0.0141  0.1516  0.0133  0.0606 -0.1009
#>  -0.0160 -0.1533 -0.0492 -0.0738  0.1385
#>  -0.1819  0.1906  0.0515  0.0980  0.1830
#>  -0.0678  0.1724  0.0914 -0.1575  0.1338
#> 
#> (3,1,.,.) = 
#>  -0.0386  0.0314  0.0473 -0.1881 -0.0848
#>  -0.0805  0.1641 -0.0009 -0.1189  0.1966
#>   0.0166  0.1321  0.0586 -0.0890 -0.0104
#>   0.1341  0.1446 -0.0266  0.1377  0.0130
#>   0.0878 -0.0120  0.1676 -0.0057 -0.0602
#> 
#> (4,1,.,.) = 
#>   0.0808  0.1956  0.0116  0.0120 -0.0445
#>  -0.0620 -0.1917  0.0159 -0.0426  0.1548
#>   0.1291  0.0993 -0.0191  0.1040  0.0930
#>  -0.0752 -0.0272 -0.1484  0.0472 -0.1651
#>   0.1708  0.0247 -0.0511  0.1567 -0.1810
#> 
#> (5,1,.,.) = 
#>   0.1985 -0.0241  0.1558  0.0839  0.1445
#> ... [the output was truncated (use n=-1 to disable)]
#> [ CPUFloatType{20,1,5,5} ][ requires_grad = TRUE ]
```
It is usual for model parameters to then be passed to an optimiser.

This can also be initialized within a 'mirai' process.

``` r
optim <- mirai(optim_rmsprop(params = params), params = m$data$parameters)

optim[]
#> <optim_rmsprop>
#>   Inherits from: <torch_optimizer>
#>   Public:
#>     add_param_group: function (param_group) 
#>     clone: function (deep = FALSE) 
#>     defaults: list
#>     initialize: function (params, lr = 0.01, alpha = 0.99, eps = 1e-08, weight_decay = 0, 
#>     load_state_dict: function (state_dict, ..., .refer_to_state_dict = FALSE) 
#>     param_groups: list
#>     state: State, R6
#>     state_dict: function () 
#>     step: function (closure = NULL) 
#>     zero_grad: function () 
#>   Private:
#>     step_helper: function (closure, loop_fun)

daemons(0)
#> [1] 0
```
Above, tensors and complex objects containing tensors were passed seamlessly between host and daemon processes, in the same way as any other R object.

The custom serialization in `mirai` leverages R's own native 'refhook' mechanism to allow such completely transparent usage. Designed to be fast and efficient, data copies are minimised and the 'official' serialization methods from the `torch` package are used directly.