nestedtune

You tune a model with cross-validation, keep the setting with the best score, and then score that setting once on a held-out test set. That test score is one number from one split of the data, and nothing in it says whether the split is representative. nestedtune scores the whole tune-and-fit procedure on several outer splits instead. The outer scores give the mean across those splits and show how far the score moves from one split to the next. Each outer fold tunes on its own inner resamples with tune or finetune, so no outer score is the score that picked its fold’s winner. The score that picks a winner tends to be optimistic, because the winner was picked for scoring well. nestedtune keeps what every fold chose.

The mean of the outer scores is the number to report for the model you deploy. That model is the same procedure run once more on all the data, so there is no second number to compute for it.

Installation

# install.packages("pak")
pak::pak("tidymodels/nestedtune")

Example

library(tidymodels)
library(nestedtune)

set.seed(1)
folds <- nested_resamples(
  mtcars,
  outside = vfold_cv(v = 5),
  inside = vfold_cv(v = 5)
)

wf <- workflow(
  mpg ~ .,
  rand_forest(mtry = tune(), min_n = tune()) |>
    set_engine("ranger") |>
    set_mode("regression")
)
grid <- expand.grid(mtry = c(2L, 5L, 8L), min_n = c(2L, 10L))

set.seed(2)
res <- nested_tune_grid(wf, folds, grid = grid)

# The number to report for the model you deploy.
collect_metrics(res)
#> # A tibble: 2 × 5
#>   .metric .estimator  mean     n std_err
#>   <chr>   <chr>      <dbl> <int>   <dbl>
#> 1 rmse    standard   2.46      5  0.445 
#> 2 rsq     standard   0.844     5  0.0267

# The model to deploy, the same procedure run once more on all the data.
set.seed(3)
final <- nested_final_fit(wf, res)
predict(final, new_data = mtcars[1:3, ])
#> # A tibble: 3 × 1
#>   .pred
#>   <dbl>
#> 1  20.9
#> 2  20.9
#> 3  23.8

Supported resampling designs

The table covers 15 resampling functions from rsample. It says whether each one works as the outer loop and as the inner loop of a nested design.

Function	Outer loop	Inner loop
vfold_cv()	Yes	Yes
mc_cv()	Yes	Yes
group_vfold_cv()	Yes	Yes
group_mc_cv()	Yes	Yes
clustering_cv()	Yes	Yes
bootstraps()	Refused	Yes
group_bootstraps()	Refused	Yes
loo_cv()	Refused	Refused
apparent()	Refused	Refused
validation_set()	Yes1	No
permutations()	Refused	Refused
rolling_origin()	Yes	Yes
sliding_window()	Yes	Yes
sliding_index()	Yes	Yes
sliding_period()	Yes	Yes

Yes means that the test suite runs the design in that loop through nested_tune_grid(), with the arguments those tests use, and that every outer fold completes. The design in the other loop is a v-fold design, or a time-series design for the four time-series rows.

Refused means that nestedtune stops with an error before any fold runs. nested_resamples() refuses such a design when it builds it, and each nestedtune function refuses one built another way, such as with rsample::nested_cv(). No means that the design cannot be built in that loop. The “Differences from rsample” section of ?nested_resamples gives the reason for each Refused and No cell. The same page also gives the rules that refuse a single split in either loop. The two racing functions also refuse an inner apparent split, as ?nested_tune_race explains.

Supported tuning functions

The table lists tuning functions from tune, finetune, workflowsets and tidyclust, and the nestedtune function that does each one’s job on a nested design.

Function	nestedtune function	Supported
tune::tune_grid()	nested_tune_grid()	Yes
tune::tune_bayes()	nested_tune_bayes()	Yes
tune::fit_resamples()	nested_fit_resamples()	Yes
finetune::tune_race_anova()	nested_tune_race_anova()	Yes
finetune::tune_race_win_loss()	nested_tune_race_win_loss()	Yes
finetune::tune_sim_anneal()	nested_tune_sim_anneal()	Yes
workflowsets::workflow_map()	nested_workflow_map()	Yes
tidyclust::tune_cluster()		No

Yes means that nestedtune exports a nested counterpart that does the row’s job on a nested design. The test suite runs it with vfold_cv() in both loops, and every outer fold completes. nested_fit_resamples() and nested_workflow_map() run no search of their own. The first scores a workflow with nothing to tune, and the second runs each workflow of a set through one of the other six functions. No means that no nestedtune function does the row’s job. For tune_cluster(), nothing yet settles what a nested estimate of a clustering metric means.

The resampling table’s Yes cells were tested through nested_tune_grid(). The “Time-series designs” section of ?nested_resamples says which time-series designs the other six functions are tested on. Choosing the inner tuner runs the Bayesian search, the two racing searches and simulated annealing on one example.

Learn more:

• Nested cross-validation, the path from a design to a write-up.
• What the estimate means, which quantity the nested number is and how far to trust it.
• Choosing the inner tuner, the Bayesian search, the two racing searches, simulated annealing and a set of workflows on one design.
• Reading the results, every column of the results object and every function that reads it.
• Running the outer loop in parallel, the same call on a pool of mirai daemons.

Footnotes

1. Build the validation set before the call, with split <- initial_validation_split(data) and validation_set(split). Then pass rbind(training(split), validation(split)) as data, because those are the rows the validation set refers to. A call such as outside = validation_set() cannot be built, because the function takes a split rather than a data frame. ↩