Setup

## install.packages("devtools")
## devtools::install_github("r-lib/devtools")
url <- "http://owos.gm.fh-koeln.de:8055/bartz/spot.git"
devtools::install_git(url = url)

• Package version SPOT is at least 2.0.21.

library("SPOT")
packageVersion("SPOT")
#> [1] '2.1.2'

Introduction

• The performance of modern search heuristics such as evolution strategies (ES), differential evolution (DE), or simulated annealing (SANN) relies crucially on their parameterizations—or, statistically speaking, on their factor settings.

• Finding good parameter settings for an optimization algorithm will be referred to as tuning.

• We will illustrate how an existing search heuristic can be tuned using the sequential parameter optimization toolbox (SPOT), which is one possible implementation of the sequential parameter optimization (SPO) framework introduced in~.

• The version of SPOT presented in this article is implemented in R.

• R is a freely available language and environment for statistical computing and graphics which provides a wide variety of statistical and graphical techniques: linear and nonlinear modelling, statistical tests, time series analysis, classification, clustering, etc."~.

• can be downloaded from CRAN.

• The package can be installed from within R using the installPackages command.

install.packages("SPOT")

• Note, that the package only has to be installed once (unless an update to a more recent version is needed).
• Unlike installation, the package has to be loaded to the R work space every time a new R session is started.
• SPOT can be loaded to the work space with R’s library command.

library("SPOT")

• In order to keep the setup as simple as possible, we will use simulated annealing for illustrating the tuning procedure~.

• Simulated annealing is freely available in .

• This implementation of the simulated annealing heuristic will be referred to as SANN in the following.

• Response surface methodology (RSM) will be used in this article~.

• It can be seen as a set of statistical methods for empirical model building.

• Using design of experiments, a response (dependent variable, output variable, or fitness value, \(y\)) that depends on one or several input variables (independent variables or solutions, \(\vec{x}\)) is optimized.

• The underlying model can be formulated as \[ y = f(\vec{x}) + \epsilon, \] where \(\epsilon\) represents some noise (uncertainty, error observed) in the response \(y\).

• The term response surface refers to the surface represented by \(f(\vec{x})\).

• In order to estimate the quality of a solution, the term fitness is used in evolutionary optimization.

• In physics, the concept of a potential or energy function is used.

• Since we are dealing with minimization, a low fitness value \(f(\vec{x})\) implies that \(\vec{x}\) is a good solution.

• This report is structured as follows.

  • The algorithm-tuning framework consists of three levels, which are useful for distinguishing several problem domains.
  • The first level (objective function) is detailed.
  • A second level (optimization algorithm) is descibed. Here, the well known simulated annealing algorithm is used. * The third level (tuning) is explained. This tuning example can be used as a starting point for beginners.
  • Details of the SPOT configuration are described.
  • Visual inspection of the results plays an important role in the SPOT approach.
  • Plot functions that are provided in the SPOT toolbox are described. These enable an exploratory fitness landscape analysis.
  • The response surface methodology is introduced.
  • Tools for an statistical analysis are described.
  • Applying SPOT to deterministic problems is briefly explained.
  • Finally, ensemble models via stacking are described.

Levels During Tuning and Optimization

• We will consider algorithm tuning and surrogate model based optimization in this article.
• The following Figure shows the different levels that are used in these scenarios.

Level L1: Objective Function

• Before SANN can be started, the user has to specify an objective function \(f\).
• To keep things as simple as possible, the sphere function \[ f(x) = \sum_{i=1}^n x_i^2 \] will be used.

sphere <- function (x){
  sum(x^2)
  }
sphere( c(1,2) )
#> [1] 5

• The sphere function uses vector inputs.
• A matrix-based implementation is defined as funSphere in the SPOT package.

funSphere
#> function (x) 
#> {
#>     matrix(apply(x, 1, function(x) {
#>         sum(x^2)
#>     }), , 1)
#> }
#> <bytecode: 0x7fd25f846438>
#> <environment: namespace:SPOT>

function (x) 
{
    matrix(apply(x, 1, function(x) {
        sum(x^2)
    }), , 1)
}

• A surface plot of this function in the interval \([0;1] \times [0;1]\) can be generated using the following command:

plotFunction(funSphere)

Level L2: Simulated Annealing SANN

• Simulated annealing is a generic probabilistic heuristic algorithm for global optimization~.

• The name comes from annealing in metallurgy.

• Controlled heating and cooling of a material reduces defects.

• Heating enables atoms to leave their initial positions (which are local minima of their internal energy), and controlled cooling improves the probability to find positions with lower states of internal energy than the initial positions.

• The SANN algorithm replaces the current solution with a randomly generated new solution.

• Better solutions are accepted deterministically, where worse solutions are accepted with a probability that depends on the difference between the corresponding function values and on a global parameter, which is commonly referred to as the temperature.

• The algorithm parameter temp specifies the initial temperature of the SANN algorithm.

• The temperature is gradually decreased during the optimization.

• A second parameter, `tmax, is used to model this cooling scheme.

• We consider the R implementation of SANN, which is available via the general-purpose optimization function optim() from the R package stats, which is part of every R installation.

• The function optim() is parametrized as follows

optim(par, fn, gr = NULL, ..., method = c("Nelder-Mead", "BFGS", "CG", "L-BFGS-B", "SANN", "Brent"), lower = -Inf, upper = Inf, control = list(), hessian = FALSE)

• Here, par denotes initial values for the parameters to be optimized over.

• Note, the problem dimension is specified by the length of this vector, so par=c(1,1,1,1) denotes a four-dimensional optimization problem.

• fn is a function to be minimized (or maximized), with first argument the vector of parameters over which minimization is to take place.

• gr defines a function to return the gradient for the BFGS, CG and L-BFGS-B methods.

• If it is NULL, a finite-difference approximation will be used.

• For the SANN method it specifies a function to generate a new candidate point.

• If it is NULL, a default Gaussian Markov kernel is used.

• The symbol ... represents further arguments (optional) that can be be passed to fn and gr.

• The parameter method denotes the optimization method to be used.

• Here, we will use the parameter value SANN.

• The parameters lower, upper specify bounds on the variables for the “L-BFGS-B” method, or bounds in which to search for method Brent.

• So, we will not use these variables in our examples.

• The argument control defines a relatively long list of control parameters}. We will use the following parameters from this list:

  1. maxit, i.e., the maximum number of iterations, which is for SANN the maximum number of function valuations. This is the stopping criterion.
  2. temp controls the SANN algorithm. It is the starting temperature for the cooling schedule with a default value of 10.
  3. Finally, we will use tmax, which is the number of function evaluations at each temperature for the SANN method. Its default value is also 10.

• To obtain reproducible results, we will set the random number generator (RNG) seed.

• Using a two-dimensional objective function (sphere) and the starting point (initial values for the parameters to be optimized over) \((10,10)\), we can execute the optimization runs as follows:

set.seed(123)
resSANN <- optim(c(10,10), sphere, method="SANN",
                control=list(maxit=100, temp=10, tmax = 10))
resSANN
#> $par
#> [1] 4.835178 4.664964
#> 
#> $value
#> [1] 45.14084
#> 
#> $counts
#> function gradient 
#>      100       NA 
#> 
#> $convergence
#> [1] 0
#> 
#> $message
#> NULL

• The best, i.e., smallest, function value, which was found by SANN, reads 45.14084.

• The corresponding point in the search space is approximately (4.835178, 4.664964).

• No gradient information was used and one hundred function evaluations were performed.

• The variable convergence is an integer code, and its value 0 indicates successful completion of the SANN run.

• No additional message is returned.

• Now that we have performed a first run of the SANN algorithm on our simple test function, we are interested in improving SANN’s performance.

• The SANN heuristic requires some parameter settings, namely temp and tmax. If these values are omitted, a default value of ten is used.

• The questions is:

  • Are the default algorithm parameter settings, namely temp=10 and tmax=10, adequate for SANN or can these values be improved?

• That is, we are trying to tune the SANN optimization algorithm.

• A typical beginner in algorithm tuning would try to improve the algorithm’s performance by manually increasing or decreasing the algorithm parameter values, e.g., choosing temp = 20 and tmax = 5.

set.seed(123)
resSANN <- optim(par = c(10,10), fn = sphere, method="SANN",
                control = list(maxit = 100, temp = 20, tmax = 5))
resSANN
#> $par
#> [1] 6.163905 6.657100
#> 
#> $value
#> [1] 82.3107
#> 
#> $counts
#> function gradient 
#>      100       NA 
#> 
#> $convergence
#> [1] 0
#> 
#> $message
#> NULL

• Obviously, the manual ``tuning" step worsened the result.
• And, this procedure is very time consuming and does not allow efficient statistical conclusions.
• Therefore, we will present a different approach, which uses SPOT.
• Although the setup for the tuning procedure with SPOT is very similar to the setup discussed in this section, it enables deeper insights into the algorithm’s performance.

Level L3: SPOT

• This section presents an example, which demonstrates, how SPOT can be used to tune the SANN algorithm defined at level L2.
• The goal of this tuning procedure is to determine improved parameter settings for the SANN algorithm.
• As a level L1 function, which will be optimized by SANN, the sphere function was chosen.

x0 =  c(-1,1,-1) 

* Since `x0` has three elements, we are facing a three dimensional optimization problem.
* `SANN` will be used to determine its minimum function value.

maxit = 100 

sann2spot <- function(algpar){
  performance <- NULL
  for (i in 1:nrow(algpar)){
    resultList <- optim(par = c(10,10),
                    fn = sphere,
                    method = "SANN",
                    control = list(maxit = 100,
                                  temp = algpar[i,1],
                                  tmax = algpar[i,2]))
    performance <- c(performance,resultList$value)
    }
return(matrix(performance,,1))
}

set.seed(123)
sann2spot(algpar = matrix(c(10,10),1))
#>          [,1]
#> [1,] 45.14084

set.seed(123)
sann2spot(algpar=matrix(c(5,20),1))
#>          [,1]
#> [1,] 4.469163

spotConfig <- list(
  types = c("integer", "integer"), #data type of tuned parameters
  funEvals = 50, #maximum number of SANN runs
  noise = TRUE, #problem is noisy (SANN is non-deterministic)
  seedFun = 1, #RNG start seed for algorithm calls (iterated)
  replicates = 2, #2 replicates for each SANN parameterization
  seedSPOT = 1, #main RNG
  design = designLHD, #initial design: Latin Hypercube
  model = buildRandomForest, # model = buildKriging Kriging surrogate model
  optimizer = optimLHD, #Use LHD to optimize on model 
  optimizerControl = list(funEvals=100) #100 model evals in each iteration  
)

The Region of Interest

• The region of interest (ROI) specifies SPOT’s search intervals for the SANN parameters, i.e., for tmax and temp.
• Here, both parameters temp and tmax will be tuned in the region between one and 100.

tempLo = 1
tempHi = 100
tmaxLo = 1
tmaxHi = 100
lower=c(tempLo,tmaxLo)
upper=c(tempHi,tmaxHi)

• The order (first temp then tmax) has to be the same as in the sann2spot() interface.

Calling spot()

• Now we are ready to perform our first SPOT experiment.
• We will start SPOT via spot().
• The result is stored in the list resRf:

resRf <- spot(x=NULL,
              fun=sann2spot,
              lower=lower,
              upper=upper,
              control=spotConfig)

• Now, we are able to take a look at the results from the tuning procedure.
• Output from the SPOT run, which is stored in the list resRf, has the following structure.

str(resRf)
#> List of 7
#>  $ xbest   : num [1, 1:2] 12 78
#>  $ ybest   : num [1, 1] 0.0085
#>  $ x       : num [1:50, 1:2] 9 72 45 15 26 57 70 93 33 88 ...
#>  $ y       : num [1:50, 1] 0.226 28.891 59.309 0.192 2.864 ...
#>  $ count   : int 50
#>  $ msg     : chr "budget exhausted"
#>  $ modelFit:List of 3
#>   ..$ rfFit:List of 17
#>   .. ..$ call           : language randomForest(x = x, y = y)
#>   .. ..$ type           : chr "regression"
#>   .. ..$ predicted      : num [1:49] 0.323 31.175 23.541 28.39 10.454 ...
#>   .. ..$ mse            : num [1:500] 226 161 223 185 180 ...
#>   .. ..$ rsq            : num [1:500] 0.364 0.547 0.371 0.478 0.492 ...
#>   .. ..$ oob.times      : int [1:49] 177 175 194 177 192 165 174 199 212 173 ...
#>   .. ..$ importance     : num [1:2, 1] 7780 7508
#>   .. .. ..- attr(*, "dimnames")=List of 2
#>   .. .. .. ..$ : chr [1:2] "1" "2"
#>   .. .. .. ..$ : chr "IncNodePurity"
#>   .. ..$ importanceSD   : NULL
#>   .. ..$ localImportance: NULL
#>   .. ..$ proximity      : NULL
#>   .. ..$ ntree          : num 500
#>   .. ..$ mtry           : num 1
#>   .. ..$ forest         :List of 11
#>   .. .. ..$ ndbigtree    : int [1:500] 29 31 31 29 33 23 25 27 31 31 ...
#>   .. .. ..$ nodestatus   : int [1:37, 1:500] -3 -1 -3 -3 -3 -1 -1 -3 -3 -3 ...
#>   .. .. ..$ leftDaughter : int [1:37, 1:500] 2 0 4 6 8 0 0 10 12 14 ...
#>   .. .. ..$ rightDaughter: int [1:37, 1:500] 3 0 5 7 9 0 0 11 13 15 ...
#>   .. .. ..$ nodepred     : num [1:37, 1:500] 8.85 65.11 3.85 20.22 1.33 ...
#>   .. .. ..$ bestvar      : int [1:37, 1:500] 2 0 2 1 1 0 0 1 1 1 ...
#>   .. .. ..$ xbestsplit   : num [1:37, 1:500] 50.5 0 70.5 49 16.5 0 0 9 40 4.5 ...
#>   .. .. ..$ ncat         : num [1:2] 1 1
#>   .. .. ..$ nrnodes      : int 37
#>   .. .. ..$ ntree        : num 500
#>   .. .. ..$ xlevels      :List of 2
#>   .. .. .. ..$ : num 0
#>   .. .. .. ..$ : num 0
#>   .. ..$ coefs          : NULL
#>   .. ..$ y              : num [1:49, 1] 0.226 28.891 59.309 0.192 2.864 ...
#>   .. ..$ test           : NULL
#>   .. ..$ inbag          : NULL
#>   .. ..- attr(*, "class")= chr "randomForest"
#>   ..$ x    : num [1:49, 1:2] 9 72 45 15 26 57 70 93 33 88 ...
#>   ..$ y    : num [1:49, 1] 0.226 28.891 59.309 0.192 2.864 ...
#>   ..- attr(*, "class")= chr "spotRandomForest"

• SPOT generates many information which can be used for a statistical analysis.
• For example, the best configuration found can be displayed as follows:

cbind(resRf$xbest, resRf$ybest)
#>      [,1] [,2]        [,3]
#> [1,]   12   78 0.008496108

• SPOT recommends using temp =

resRf$xbest[1]
#> [1] 12

and tmax =

resRf$xbest[2] 
#> [1] 78

• These parameter settings differ significantly from the default values.
  • These values also make sense: the starting temperature temp should be low and the number of evaluations at each temperature tmax should be high.
  • Hence, worse solutions will rarely be accepted by SANN, which leads to a very localized search.
• This is a good configuration for this example, since the sphere function is unimodal.

Details 1: The spot() Interface

Details 2: SPOT Configuration

• SPOT configuration settings are listed in following Table.

• The configuration list stores information about SPOT specific settings.
• All settings not specified in the list will be set to their default values.

Details 3: Initial Designs

• The problem design comprehends the information about the optimization problem, e.g., the starting point x0 = (-1,1,-1) belongs to the problem design.
• A region of interest (ROI) specifies algorithm parameters and associated lower and upper bounds for the algorithm parameters.
  • Values for temp are chosen from the interval \([1; 100]\).
  • Values for tmax are chosen from the interval \([1; 100]\).
• SPOT implements a sequential approach, i.e., the available budget is not used in one step. Rather, sequential steps are made, comprised of model training, optimization, and evaluation. To initialize this procedure, some first data set is required to train the first, coarse-grained meta model.

spotConfig10 <- list(
  funEvals = 10,
  designControl = list(
    size = 6,
    replicates = 1
  ),
  noise = TRUE,
  seedFun = 1,
  seedSPOT = 1,
  replicates = 2,
  model = buildRandomForest 
)

res10 <- spot( ,fun=sann2spot
              ,lower=lower
              ,upper=upper
              ,control=spotConfig10)

cbind(res10$x, res10$y)
#>            [,1]      [,2]         [,3]
#>  [1,]  8.954322 63.418391   0.29353146
#>  [2,] 93.392836 43.125099 107.32848102
#>  [3,] 25.643432 26.240373  16.14928784
#>  [4,] 70.072590 96.524378  11.29012282
#>  [5,] 47.651660 67.384965   3.93335321
#>  [6,] 61.529701  8.874296  73.12225819
#>  [7,]  6.561794 81.375416   0.25116700
#>  [8,]  6.561794 81.375416   0.02126833
#>  [9,] 11.225956 75.246159   2.66469591
#> [10,] 11.225956 75.246159   0.07434020

designLHD(,-1,1) 
#>             [,1]
#>  [1,]  0.4350726
#>  [2,] -0.2725918
#>  [3,]  0.9844751
#>  [4,]  0.1061902
#>  [5,] -0.9456038
#>  [6,] -0.5142940
#>  [7,]  0.6935275
#>  [8,]  0.2901311
#>  [9,] -0.0977685
#> [10,] -0.6617153

designLHD(, c(-1,-2,1,0),c(1,4,9,1)
          , control=list(size=5, retries=100, types=c("numeric","integer","factor","factor")))
#>             [,1] [,2] [,3] [,4]
#> [1,] -0.00173205    3    6    0
#> [2,]  0.41823095   -1    9    1
#> [3,] -0.45046368    4    5    1
#> [4,] -0.82865697    1    1    1
#> [5,]  0.68113235   -2    3    0

set.seed(123)
x1 <- designLHD(,c(-1,-1),c(1,1),control=list(size=50,retries=100))
x2 <- designLHD(x1,c(-2,-2),c(2,2),control=list(size=50,retries=100))

plot(x2,pch=1)
points(x1, pch=4)

designUniformRandom(,c(-1,0),c(1,10),control=list(size=5))
#>            [,1]     [,2]
#> [1,]  0.3480606 3.447254
#> [2,] -0.3917176 2.413147
#> [3,]  0.7143919 8.768452
#> [4,] -0.4602907 1.681719
#> [5,] -0.5792664 9.799514

Details 4: Models

• In SPOT, a meta model or surrogate model is used to determine promising algorithm design points.
• To that end, it aims to learn the relation between algorithm parameters and the corresponding algorithm performance. * The different models and their options are described next.

# Objective function
braninFunction <- function (x) {
    (x[2]  - 5.1/(4 * pi^2) * (x[1] ^2) + 5/pi * x[1]  - 6)^2 + 10 * (1 - 1/(8 * pi)) * cos(x[1] ) + 10
}
## Create 20 design points
set.seed(1)
x <- cbind(runif(20)*15-5, runif(20)*15)
## Compute observations at design points (for Branin function)
y <- as.matrix(apply(x,1,braninFunction))
## Create model with default settings
fit <- buildKriging(x,y,control = list(algTheta=optimLHD))
## Print model parameters
print(fit)
#> ------------------------
#> Forrester Kriging model.
#> ------------------------
#> Estimated activity parameters (theta) sorted 
#> from most to least important variable 
#> x1  x2 
#> 7.575502 1.361329
#>  
#> exponent(s) p:
#> 2
#>  
#> Estimated regularization constant (or nugget) lambda:
#> 5.239774e-06
#>  
#> Number of Likelihood evaluations during MLE:
#> 600
#> ------------------------
##Define a new location
newloc <- matrix(c(1,2),nrow =1 )
##Predict at new location
predict(fit,newloc)
#> $y
#> [1] 21.08753
## True value at location
braninFunction(newloc)
#> [1] 21.62764
## 

Handling Factor Variables in the Kriging Model

• Sometimes, parameters optimized or modeled by SPOT functions will not be numerical, but rather categorical.
• This may, e.g., occur if an evolutionary algorithm is tuned:
  • while some parameters like mutation rates may be real valued, the selection between different mutation operators may be a categorical parameter.
• Hence, if \(x_i\), the \(i\)th dimension of a parameter configuration \(\vec{x}\), is a factor variable (see parameter types), Hamming distance, that determines the number of positions at which the corresponding values are different, will be used instead of \(|x_i-x'_i|\).
• To illustrate how factor variables can be handled, we create a test function that uses a factor variable.
  • Here, the third dimension \(x_3\) is categorical.

braninFunctionFactor <- function (x) {  
y <- (x[2] - 5.1 / (4 * pi^2) * (x[1]^2) + 5 / pi * x[1] - 6)^2 + 10 * (1 - 1 / (8 * pi)) * cos(x[1]) + 10
    if(x[3] == 1)
        y <- y + 1
    else if(x[3]==2)
        y <- y - 1
    y  
}

• To test how this affects our model, we first generate some training data and fit the model with default settings, which ignores factor information and uses the standard kernel.

set.seed(1)
## Replace x with new data
x <- cbind(runif(50)*15-5,runif(50)*15,sample(1:3,50,replace=TRUE))
##
y <- as.matrix(apply(x,1,braninFunctionFactor))
fitDefault <- buildKriging(x,y,control = list(algTheta=optimLBFGSB))

• Afterwards we fit the model, which includes information about the factor variable.

fitFactor <- buildKriging(x,y,control = list(algTheta=optimLBFGSB,types=c("numeric","numeric","factor")))

• We generate some new, unseen data for testing and perform some predictions with both models.

##Replace xtest with new data
xtest <- cbind(runif(200)*15-5,runif(200)*15,sample(1:3,200,replace=TRUE))
##
ytest <- as.matrix(apply(xtest,1,braninFunctionFactor))
## Predict test data with both models, and compute error
ypredDef <- predict(fitDefault,xtest)$y
ypredFact <- predict(fitFactor,xtest)$y
mean((ypredDef-ytest)^2)
#> [1] 4.099175
mean((ypredFact-ytest)^2)
#> [1] 2.094953

• The error of the factor-aware model is lower.
• This demonstrates that users should make sure to declare the nature of the modeled variables correctly, via the types variable.

Details 5: Optimization on the Meta Model

• Minimization by Latin hyper cube sampling (LHS) is the default optimizer used for finding the next algorithm design parameters on the meta model.
• The LHS procedure generates a set of LHD points.
• The function optimLHD() uses LHS to optimize a specified target function as follows:
  • A Latin hyper cube Design (LHD) is created with designLHD(), then evaluated by the objective function.
  • All results are reported, including the best (minimal) objective value, and corresponding design point.
• A standalone optimization run using optimLHD() can be implemented as follows. It uses 100 design points as a default value.

resOptimumLHD <- optimLHD(,fun = funSphere,lower = c(-10,-20),upper=c(20,8))
str(resOptimumLHD)
#> List of 6
#>  $ x    : num [1:100, 1:2] -5.528 -6.509 0.718 -4.327 15.608 ...
#>  $ y    : num [1:100, 1] 48.8 62.5 295.3 260.7 366.2 ...
#>  $ xbest: num [1, 1:2] 0.136 -0.558
#>  $ ybest: num [1, 1] 0.33
#>  $ count: num 100
#>  $ msg  : chr "success"
resOptimumLHD$ybest
#>         [,1]
#> [1,] 0.32992

• Using more sophisticated algorithms, as the variable metric algorithm (L-BFGS-B), might lead to better results.
• However, they are not as robust as the simple optimLHD() search.
• Also, L-BFGS-B is a pure local search, which may not be ideal to solve potentially multi-modal tuning problems.

resOptimBFGS <- optimLBFGSB(,fun = funSphere,lower = c(-10,-20),upper=c(20,8))
resOptimBFGS$ybest
#> [1] 2.098584e-40

• Hence, SPOT also includes interfaces to more sophisticated algorithms, such as differential evolution from DEoptim package or various methods included in the nloptr package.
• For further details on these, see e.g., the help of the corresponding functions.

Details 6: SPOT Loop: Continued Evaluation

• Sometimes, users may desire to continue a previously finished SPOT run.
• We will demonstrate, how SPOT can be restarted, reusing the collected data.

control01 <- list(
    designControl = list(size = 5, 
                       replicates = 1),    
    funEvals = 5)
res1 <- spot(,funSphere,
             lower = c(-2,-3),
             upper = c(1,2),
             control01)
cbind(res1$x, res1$y)
#>            [,1]       [,2]     [,3]
#> [1,] -0.8963358  0.2026923 0.844502
#> [2,] -1.7919899 -1.2888788 4.872436
#> [3,]  0.6002650 -0.8783081 1.131743
#> [4,] -0.5141893 -2.7545115 7.851724
#> [5,]  0.3353190  1.1433044 1.419584

control01$funEvals <- 8
res2 <- spotLoop(res1$x,
                 res1$y,
                 funSphere,
                 lower = c(-2,-3),
                 upper = c(1,2),
                 control01)
cbind(res2$x, res2$y)
#>            [,1]         [,2]      [,3]
#> [1,] -0.8963358  0.202692255 0.8445020
#> [2,] -1.7919899 -1.288878778 4.8724363
#> [3,]  0.6002650 -0.878308079 1.1317431
#> [4,] -0.5141893 -2.754511486 7.8517241
#> [5,]  0.3353190  1.143304379 1.4195837
#> [6,]  0.7868298 -0.009795371 0.6191971
#> [7,]  0.4923921 -0.124161024 0.2578659
#> [8,]  0.3555719  0.023539907 0.1269855

Introduction to SPOT’s Plot Functions

• The SPOT package offers three plot functions that can be used to visualize data or evaluate a model’s performance creating 2D and 3D surface plots.
  • plotFunction() plots function objects
  • plotData() plots data
  • plotModel() plots model objects, created by build* functions from the SPOT package.

plotFunction(funSphere, rep(-1,2), rep(1,2)) 

myFunction <- function (x){
  matrix(apply(x, # matrix
               1, # margin (apply over rows)
               function(x) sum(x^3-1) # objective function
               ),
         , 1) # number of columns
  }
plotFunction(myFunction, 
             rep(-1,2), 
             rep(1,2), 
             color.palette = rainbow) 

plotFunction(myFunction,
             rep(-1,2), 
             rep(1,2), 
             type="persp", 
             theta=10, 
             phi=25, 
             border = NA)

set.seed(123)
k <- 30
x.test <- designLHD(,rep(-1,3),rep(1,3), control = list(size = k)) 
y.test <- funSphere(x.test)
head( cbind(x.test, y.test))
#>            [,1]        [,2]        [,3]      [,4]
#> [1,]  0.6721562 -0.67693655 -0.06068276 0.9137194
#> [2,] -0.0813375  0.74944336  0.59109728 0.9176771
#> [3,]  0.5462156 -0.39765660 -0.09992404 0.4664671
#> [4,]  0.2437057 -0.29365065  0.21488375 0.1917982
#> [5,] -0.3043235  0.25574608 -0.30583872 0.2515562
#> [6,]  0.4131432  0.02414995  0.12575375 0.1870845

fit.test <- buildRSM(x.test,y.test)
plotModel(fit.test)

plotModel(fit.test,which=c(1,3),type="contour",pch1=24,col1="blue")

plotModel(fit.test,which=c(1,3),type="persp",border="NA",theta=255,phi=20)

plotData(x.test,y.test)

plotData(x.test,y.test,type="filled.contour",cex1=1,col1="red",pch1=21,model=buildRandomForest)

plotData(x.test,y.test,type="persp",border=NA,model=buildLOESS)

Plotting the Final Model with plotModel()

• Plotting the final model from the SPOT run might be the most generic way of visualizing the results, because during the optimization, the optimizer trusted this model. So, why should it be considered unreliable after the optimization is finished?
• Based on the tuning results from the previoius example (using the resRf data), we will demonstrate how the final model, which was built during the SPOT run, can be plotted.
• Since the model is stored in the result list from the SPOT run, i.e., in resRf, the parameter resRf$modelFit() can be passed as an argument to the plotModel() function.
• The result is shown in the following Figure. It shows a surface plot from the SPOT run with random forest.

plotModel(resRf$modelFit)

plotData(resRf$x,resRf$y,model=buildLOESS)

plotData(resRf$x,resRf$y,model=buildKriging)

spotConfig$model = buildKriging
spotConfig$optimizer = optimLBFGSB
spotConfig$modelControl = list(algTheta=optimLBFGSB)
## Run SPOT
resK <- spot(x=NULL,
             fun=sann2spot,
             lower=lower,
             upper=upper,
             control=spotConfig)

plotModel(resK$modelFit)

spotConfig$funEvals <- 100
spotConfig$model <- buildRandomForest
res100Rf <- spotLoop(resRf$x,
                     resRf$y,
                     fun=sann2spot,
                     lower=lower,
                     upper=upper,
                     control=spotConfig)

spotConfig$model = buildKriging
spotConfig$optimizer = optimLBFGSB
spotConfig$modelControl = list(algTheta=optimLBFGSB)
res100K <- spotLoop(resK$x,
                    resK$y,
                    fun=sann2spot,
                    lower=lower,
                    upper=upper,
                    control=spotConfig)

plotModel(res100Rf$modelFit)

plotModel(res100K$modelFit)

Example: Path of the steepest descent

• First, we create some design points and compute observations at design points.
• Then, using buildRSM(), the response surface model is build.
• The function descentSpotRSM() returns the path of the steepest descent.

x <- designUniformRandom(lower=rep(-5,2), 
                         upper=rep(15,2), 
                         control=list(size=20))
y <- funSphere(x)

• Create model with default settings

fit <- buildRSM(x,y)

• Predict new point

predict(fit,cbind(1,2))
#> $y
#>      [,1]
#> [1,]    5

• True value at location:

sphere(c(1,2))
#> [1] 5

descentSpotRSM(fit)
#> Path of steepest descent from ridge analysis:
#> $x
#>          V1       V2
#> 1  5.801230 4.503401
#> 2  5.252816 3.839109
#> 3  4.674758 3.223662
#> 4  4.074467 2.647291
#> 5  3.444532 2.119765
#> 6  2.792364 1.631315
#> 7  2.125374 1.172172
#> 8  1.443562 0.752105
#> 9  0.746928 0.371114
#> 10 0.035472 0.019430
#> 
#> $y
#>               [,1]
#>  [1,] 53.934890080
#>  [2,] 42.330833844
#>  [3,] 32.245359049
#>  [4,] 23.609430973
#>  [5,] 16.358204354
#>  [6,] 10.458485338
#>  [7,]  5.891201837
#>  [8,]  2.649533179
#>  [9,]  0.695627038
#> [10,]  0.001635788

• This situation is illustrated in the following Figure, which illustrates the response surface, which fits the sphere function.

plot(fit)

RSM and SPOT

• We can use RSM for an interactive tuning approach.
• The starting point in this example are the 100 design points that were generated during the Kriging-based SPOT run in Example res100K.
• These 100 data points are used to build a response surface with buildRSM().

rsm100K <- buildRSM(x=res100K$x,
                    y=res100K$y) 
summary(rsm100K$rsmfit)
#> 
#> Call:
#> rsm(formula = y ~ FO(x1, x2) + TWI(x1, x2) + PQ(x1, x2), data = codedData)
#> 
#>             Estimate Std. Error t value  Pr(>|t|)    
#> (Intercept)  27.8212     4.2690  6.5171 3.533e-09 ***
#> x1           26.9435     2.8975  9.2990 5.689e-15 ***
#> x2          -21.5854     4.1993 -5.1402 1.488e-06 ***
#> x1:x2       -23.8875     5.9749 -3.9980 0.0001271 ***
#> x1^2         -2.2793     5.3683 -0.4246 0.6721072    
#> x2^2          1.5927     6.0336  0.2640 0.7923808    
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Multiple R-squared:  0.6211, Adjusted R-squared:  0.601 
#> F-statistic: 30.82 on 5 and 94 DF,  p-value: < 2.2e-16
#> 
#> Analysis of Variance Table
#> 
#> Response: y
#>             Df  Sum Sq Mean Sq F value    Pr(>F)
#> FO(x1, x2)   2 31156.8 15578.4 68.6460 < 2.2e-16
#> TWI(x1, x2)  1  3737.4  3737.4 16.4687 0.0001022
#> PQ(x1, x2)   2    77.3    38.7  0.1704 0.8436197
#> Residuals   94 21332.2   226.9                  
#> Lack of fit 30  3761.3   125.4  0.4567 0.9897826
#> Pure error  64 17570.9   274.5                  
#> 
#> Stationary point of response surface:
#>         x1         x2 
#> -0.7345253  1.2681073 
#> 
#> Stationary point in original units:
#>        V1        V2 
#>  14.00826 106.46699 
#> 
#> Eigenanalysis:
#> eigen() decomposition
#> $values
#> [1]  11.75633 -12.44293
#> 
#> $vectors
#>          [,1]       [,2]
#> x1  0.6480722 -0.7615789
#> x2 -0.7615789 -0.6480722

• Following the path of the steepest descent on the RSM meta model, we obtain a new design point, which can be evaluated.

(xSteep <- descentSpotRSM(rsm100K) )
#> Path of steepest descent from ridge analysis:
#> $x
#>        V1      V2
#> 1  46.080 50.2900
#> 2  41.964 52.8010
#> 3  37.456 54.8005
#> 4  32.605 56.0560
#> 5  27.264 56.1955
#> 6  21.727 55.1260
#> 7  16.190 53.0800
#> 8  10.947 50.4760
#> 9   5.900 47.5930
#> 10  1.098 44.6170
#> 
#> $y
#>            [,1]
#>  [1,] 24.476384
#>  [2,] 21.347709
#>  [3,] 18.384709
#>  [4,] 15.611535
#>  [5,] 12.920574
#>  [6,] 10.279201
#>  [7,]  7.555571
#>  [8,]  4.742900
#>  [9,]  1.725641
#> [10,] -1.472259

• We have chosen the eighth point, i.e.,

xNew <- xSteep$x[8,]

• Then we determine its function value.

(yNew <- sann2spot(xNew))
#>            [,1]
#> [1,] 0.04315271

• Next, we can refine the rsm meta model by including this point to the set of design points.

x101 <- rbind(res100K$x, xNew)
y101 <- rbind(res100K$y, yNew)
rsm101K <- buildRSM(x=x101,
                    y=y101) 
summary(rsm101K$rsmfit)
#> 
#> Call:
#> rsm(formula = y ~ FO(x1, x2) + TWI(x1, x2) + PQ(x1, x2), data = codedData)
#> 
#>             Estimate Std. Error t value  Pr(>|t|)    
#> (Intercept)  27.6726     4.2214  6.5553 2.868e-09 ***
#> x1           27.0095     2.8757  9.3922 3.291e-15 ***
#> x2          -21.6066     4.1787 -5.1706 1.291e-06 ***
#> x1:x2       -23.9811     5.9386 -4.0382 0.0001092 ***
#> x1^2         -2.1542     5.3273 -0.4044 0.6868473    
#> x2^2          1.7749     5.9759  0.2970 0.7671137    
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Multiple R-squared:  0.6218, Adjusted R-squared:  0.6018 
#> F-statistic: 31.23 on 5 and 95 DF,  p-value: < 2.2e-16
#> 
#> Analysis of Variance Table
#> 
#> Response: y
#>             Df  Sum Sq Mean Sq F value    Pr(>F)
#> FO(x1, x2)   2 31260.8 15630.4 69.5376 < 2.2e-16
#> TWI(x1, x2)  1  3763.0  3763.0 16.7412 8.982e-05
#> PQ(x1, x2)   2    77.6    38.8  0.1725    0.8418
#> Residuals   95 21353.7   224.8                  
#> Lack of fit 31  3782.8   122.0  0.4445    0.9924
#> Pure error  64 17570.9   274.5                  
#> 
#> Stationary point of response surface:
#>        x1        x2 
#> -0.715249  1.254782 
#> 
#> Stationary point in original units:
#>       V1       V2 
#>  14.9528 105.8474 
#> 
#> Eigenanalysis:
#> eigen() decomposition
#> $values
#> [1]  11.96075 -12.34012
#> 
#> $vectors
#>          [,1]       [,2]
#> x1  0.6474238 -0.7621302
#> x2 -0.7621302 -0.6474238

• The resulting plot is shown in the following Figure.

plot(rsm101K)

• Now, we have two options for continuing the optimization.
• Either, we continue following the path of the steepest descent, i.e., we use SPOT’s descentSpotRSM() function.

descentSpotRSM(rsm101K)
#> Path of steepest descent from ridge analysis:
#> $x
#>        V1      V2
#> 1  46.080 50.2900
#> 2  41.915 52.8010
#> 3  37.456 54.8005
#> 4  32.556 56.0095
#> 5  27.264 56.1490
#> 6  21.678 55.0330
#> 7  16.190 52.9870
#> 8  10.898 50.3830
#> 9   5.900 47.5465
#> 10  1.098 44.5240
#> 
#> $y
#>            [,1]
#>  [1,] 24.323160
#>  [2,] 21.168389
#>  [3,] 18.232360
#>  [4,] 15.451965
#>  [5,] 12.788653
#>  [6,] 10.133240
#>  [7,]  7.438115
#>  [8,]  4.600924
#>  [9,]  1.619128
#> [10,] -1.570104

• Or, we can continue with SPOT.
• Following this second option, we have built an updated Kriging model using nine additional function evaluations.

spotConfig$model = buildKriging
spotConfig$optimizer = optimLBFGSB
spotConfig$modelControl = list(algTheta=optimLBFGSB)
spotConfig$funEvals <- 110
res110K <- spotLoop(x=x101,
                    y=y101,
                    fun=sann2spot,
                    lower=lower,
                    upper=upper,
                    control=spotConfig)

• Finally, we can plot the updated model.
  • buildRSM() was used to build a response surface.
  • Data from the Kriging-based SPOT run with 100 function evaluations, one additional point, which was calculated using the steepest descent function descentSpotRSM(), and one additional SPOT run with nine additional design points, were used to generate this model.
  • Altogether, 110 design points were used to generate this model.

plotModel(res110K$modelFit)

Tree-based Analysis

• SPOT’s buildRandomForest() function is a wrapper function for the randomForest() function from the randomForest package.
• Since the randomForest package has no default plot function, we switch to the party package.
  • This package provides the ctree() function, which can be applied as follows:
• Tree based analysis with Kriging data. tmax and temp.

tmaxtempz.df <- data.frame(res100K$x[,1], res100K$x[,2], res100K$y)
names(tmaxtempz.df) <- c("tmax", "temp", "y")
tmaxtempz.tree <- party::ctree(y ~ ., data=tmaxtempz.df)
plot(tmaxtempz.tree, type="simple")

Regression Analysis

xyz100K.df <- data.frame(res100K$x[,1], res100K$x[,2], res100K$y)
names(xyz100K.df) <- c("x", "y", "z")
lm100K <- lm(z ~ x*y, data=xyz100K.df)
summary(lm100K)
#> 
#> Call:
#> lm(formula = z ~ x * y, data = xyz100K.df)
#> 
#> Residuals:
#>     Min      1Q  Median      3Q     Max 
#> -58.767  -1.434   0.318   0.392  48.363 
#> 
#> Coefficients:
#>              Estimate Std. Error t value Pr(>|t|)    
#> (Intercept) -3.077837   6.970272  -0.442    0.660    
#> x            1.063641   0.154412   6.888 5.87e-10 ***
#> y            0.039537   0.117752   0.336    0.738    
#> x:y         -0.010534   0.002573  -4.094 8.85e-05 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Residual standard error: 14.93 on 96 degrees of freedom
#> Multiple R-squared:  0.6197, Adjusted R-squared:  0.6079 
#> F-statistic: 52.15 on 3 and 96 DF,  p-value: < 2.2e-16

plot(lm100K)

par(mfrow=c(1,2))
termplot(lm100K, partial = TRUE, smooth = panel.smooth, ask=FALSE) 
par(mfrom=c(1,1))

par(mfrow=c(1,3))
car::avPlots(lm100K,ask=F)   
par(mfrow=c(1,1))