Cram Bandit Simulation

What is cram_bandit_sim()?

The cram_bandit_sim() function runs on-policy simulation for contextual bandit algorithms using the Cram method. It evaluates the statistical properties of policy value estimates such as:

• Empirical Bias on Policy Value
• Average relative error on Policy Value
• RMSE using relative errors on Policy Value
• Empirical Coverage of Confidence Intervals

This is useful for benchmarking bandit policies under controlled, simulated environments.

📋 Inputs

You need to provide:

• bandit:
  A contextual bandit environment object that generates contexts (feature vectors) and rewards for each arm.
  Example: ContextualLinearBandit, or any object following the contextual package interface.

• policy:
  A policy object that takes in a context and selects an arm (action) at each timestep.
  Example: BatchContextualLinTSPolicy, or any compatible contextual package policy.

• horizon:
  An integer specifying the number of timesteps (rounds) per simulation.
  Each simulation will run for exactly horizon steps.

• simulations:
  An integer specifying the number of independent Monte Carlo simulations to perform.
  Each simulation will independently reset the environment and policy.

• Optional Parameters:

  • alpha:
    A numeric value between 0 and 1 specifying the significance level for confidence intervals when calculating empirical coverage.
    Default: 0.05 (for 95% confidence intervals).

  • seed:
    An optional integer to set the random seed for reproducibility.
    If NULL, no seed is set.

  • do_parallel:
    A logical value indicating whether to parallelize the simulations across available CPU cores.
    Default: FALSE (parallelization disabled).
    We recommend keeping do_parallel = FALSE unless necessary, as parallel execution can make it harder for the underlying contextual package to reliably track simulation history.
    In particular, parallel runs may cause missing or incomplete entries in the recorded history, which are then discarded during analysis.

Example: Cram Bandit Simulation

# Number of time steps
horizon       <- 500L

# Number of simulations 
simulations   <- 100L

# Number of arms
k = 4

# Number of context features
d= 3

# Reward beta parameters of linear model (the outcome generation models, one for each arm, are linear with arm-specific parameters betas)
list_betas <- cramR::get_betas(simulations, d, k)

# Define the contextual linear bandit, where sigma is the scale of the noise in the outcome linear model
bandit        <- cramR::ContextualLinearBandit$new(k = k, d = d, list_betas = list_betas, sigma = 0.3)

# Define the policy object (choose between Contextual Epsilon Greedy, UCB Disjoint and Thompson Sampling)
policy <- cramR::BatchContextualEpsilonGreedyPolicy$new(epsilon=0.1, batch_size=5)
# policy <- cramR::BatchLinUCBDisjointPolicyEpsilon$new(alpha=1.0, epsilon=0.1, batch_size=1)
# policy <- cramR::BatchContextualLinTSPolicy$new(v = 0.1, batch_size=1)


sim <- cram_bandit_sim(horizon, simulations,
                            bandit, policy,
                            alpha=0.05, do_parallel = FALSE)
#> Simulation horizon: 500
#> Number of simulations: 101
#> Number of batches: 1
#> Starting main loop.
#> Finished main loop.
#> Completed simulation in 0:00:04.500
#> Computing statistics.

What Does It Return?

The output contains:

A data.table with one row per simulation, including:

• estimate: estimated policy value
  
• variance_est: estimated variance
  
• estimand: true policy value (computed from held-out context data)
  
• prediction_error: estimate - estimand
  
• est_rel_error: relative error on estimate
  
• variance_prediction_error: relative error on variance
  
• ci_lower, ci_upper: bounds of the confidence interval
  
• std_error: standard error

Result tables (raw and interactive), reporting:

• Empirical Bias on Policy Value
• Average relative error on Policy Value
• RMSE using relative errors on Policy Value
• Empirical Coverage of Confidence Intervals

Example Output Preview

head(sim$estimates)
#>      sim  estimate variance_est  estimand prediction_error est_rel_error
#>    <int>     <num>        <num>     <num>            <num>         <num>
#> 1:     1 0.5934946  0.007485342 0.5008213      0.092673298    0.18504265
#> 2:     2 0.5738572  0.004488658 0.5472302      0.026626936    0.04865765
#> 3:     3 0.3082747  0.005971414 0.2685309      0.039743854    0.14800479
#> 4:     4 0.4875583  0.001785894 0.5025160     -0.014957724   -0.02976567
#> 5:     5 0.6279068  0.002812175 0.6234795      0.004427278    0.00710092
#> 6:     6 0.6756592  0.001788615 0.6278958      0.047763481    0.07606913
#>    variance_prediction_error  std_error  ci_lower  ci_upper
#>                        <num>      <num>     <num>     <num>
#> 1:                -0.1543576 0.08651787 0.4239227 0.7630665
#> 2:                -0.4929024 0.06699744 0.4425446 0.7051697
#> 3:                -0.3253907 0.07727493 0.1568186 0.4597308
#> 4:                -0.7982420 0.04225984 0.4047305 0.5703860
#> 5:                -0.6822998 0.05302995 0.5239700 0.7318436
#> 6:                -0.7979345 0.04229202 0.5927684 0.7585501

sim$interactive_table

Notes

• list_betas is updated internally to track the true parameters per simulation
  
• The first simulation is discarded by design (due to writing issues in contextual) even when do_parallel = FALSE.

Recommended Use Cases

• Validate bandit policies under repeated experiments
  
• Compare bias and variance of different policy types
  
• Analyze empirical coverage of confidence intervals
  
• Stress-test policies under different batch sizes, sigma levels, or dimensions

References

This simulation builds on:

• Contextual bandits (contextual package)
  
• On-policy Cram estimation