Goal
Our goal for this exercise sheet is to learn how we can tune a
nonlinear pipeline consisting of multiple PipeOps generated
with mlr3pipelines. The underlying mechanism is that we
transform our pipeline into a proper learner. As such we can tune the
hyperparameters of the learner jointly with the hyperparameters of each
preprocessing step.
German Credit Dataset (dirty)
We use a dirty version of the German credit dataset of Prof. Dr. Hans Hoffman of the University of Hamburg in 1994, which contains 1000 datapoints reflecting bank customers. The dataset is available at the UCI repository as Statlog (German Credit Data) Data Set. We artificially introduced missing values to the numeric features.
library("mlr3verse")
library("data.table")
task = tsk("german_credit")
dt = task$data()
set.seed(2023)
dt = dt[sample(task$row_ids, size = 100), duration := NA]
dt = dt[sample(task$row_ids, size = 100), amount := NA]
dt = dt[sample(task$row_ids, size = 100), age := NA]
task = as_task_classif(dt, id = "german_credit_NA", target = task$target_names)
task$missings()
## credit_risk age amount credit_history
## 0 100 100 0
## duration employment_duration foreign_worker housing
## 100 0 0 0
## installment_rate job number_credits other_debtors
## 0 0 0 0
## other_installment_plans people_liable personal_status_sex present_residence
## 0 0 0 0
## property purpose savings status
## 0 0 0 0
## telephone
## 0Exercises:
Below, we built a GraphLearner which consists of a ML
pipeline that first preprocesses the data by imputing the missing values
(with one of three possible imputation methods: constant mean
imputation, random sampling imputation, and model-based imputation using
the decision tree regr.rpart), filtering features according
to the information gain, and then applies a random forest
classif.ranger learner. These steps can be reflected in the
following ML pipeline (we use branching to select only one imputation
method):
library("mlr3verse")
filter = po("filter", filter = flt("information_gain"))
impute = list(
"imputemean" = po("imputemean"),
"imputesample" = po("imputesample"),
"imputerpart" = po("imputelearner", learner = lrn("regr.rpart"))
)
ranger = lrn("classif.ranger", num.trees = 100)
graph = ppl("branch", impute) %>>% filter %>>% ranger
# Visualize the ML pipeline graph
plot(graph)
Note that the GraphLearner combines also the
hyperparameters of the learner and all other preprocessing methods. You
can run the code above as we want use it in order to automatically tune
the hyperparameters of the GraphLearner and benchmark it
with some other learners.
Create a Search Space
The elements of the above graph have different hyperparameters which
can be tuned (see the output of glrn$param_set for the
names of the hyperparameters). Set up a search space for
- the number of features to filter
information_gain.filter.nfeatby allowing values between2Land20L - the imputation method
branch.selectionby allowing all 3 values:imputemean,imputesample,imputerpart
Hint 1:
The names of the hyperparameters could be extracted from
graph$param_set. Use ps() to create a search
space. Use p_int() to define the search range for integer
hyperparameters as required by
information_gain.filter.nfeat and p_fct() for
categorical hyperparameters as required by
branch.selection.
Find the best Hyperparameters
Use the defined search space to automatically tune the number of
features for filtering and the imputation method of the
GraphLearner by setting up an AutoTuner object
with
- grid search as the tuner, with a resolution of 8, meaning that if possible up to 8 equidistant values are produced per hyperparameter
- the classification error
msr("classif.ce")as performance measure - 3-fold CV as resampling strategy
Set a seed for reproducibility (e.g.,
set.seed(2023)).
Recap AutoTuner:
The AutoTuner has the advantage over the tuning via
TuningInstanceSingleCrit or
TuningInstanceMultiCrit that we do not need to extract
information on the best hyperparameter settings at the end. Instead, the
learner is automatically trained on the whole dataset with the best
hyperparameter setting after tuning.
The AutoTuner wraps a learner and augments it with an
automatic tuning process for a given set of hyperparameters. Because the
AutoTuner itself inherits from the Learner base class, it can be used
like any other learner. The only difference is that train()
triggers the whole tuning process.
Hint 1:
With auto_tuner() a new AutoTuner instance
can be initialized. The initialization method requires the following as
an input
- the
GraphLearnerfrom the previous exercise - the hyperparameter search space (which we have already set up)
- a resampling instance initialized with
rsmp()- the 3-fold cross-validation - a performance measure - the classification error
- a termination criterion, which is
trm("none")in our case since we specify the number of resolutions in the tuner - the tuner, i.e., grid search with its
resolution.
Hint 2:
Solution:
Click me:
set.seed(2023)
glrn_tuned = auto_tuner(glrn,
resampling = rsmp("cv", folds = 3L),
search_space = tune_ps,
measure = msr("classif.ce"),
terminator = trm("none"),
tuner = tnr("grid_search", resolution = 8, batch_size = 4))
glrn_tuned$train(task)
## INFO [08:15:29.883] [bbotk] Starting to optimize 2 parameter(s) with '<OptimizerBatchGridSearch>' and '<TerminatorNone>'
## INFO [08:15:29.944] [bbotk] Evaluating 4 configuration(s)
## INFO [08:15:37.688] [bbotk] Result of batch 1:
## INFO [08:15:37.697] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:15:37.697] [bbotk] 10 imputesample 0.26500 0 0 1.63
## INFO [08:15:37.697] [bbotk] 15 imputesample 0.27599 0 0 2.69
## INFO [08:15:37.697] [bbotk] 18 imputesample 0.27301 0 0 1.56
## INFO [08:15:37.697] [bbotk] 20 imputemean 0.25701 0 0 1.19
## INFO [08:15:37.697] [bbotk] uhash
## INFO [08:15:37.697] [bbotk] b6b7275a-cfeb-4aeb-a10d-c592a760fd48
## INFO [08:15:37.697] [bbotk] 986ee86f-aab4-48d1-807c-6ca2c0bdd8c3
## INFO [08:15:37.697] [bbotk] 8d7c399e-6b28-4871-8fab-2c51d5147f60
## INFO [08:15:37.697] [bbotk] 8caab510-bae7-4a34-ba48-faa7ee4ce625
## INFO [08:15:37.716] [bbotk] Evaluating 4 configuration(s)
## INFO [08:15:46.414] [bbotk] Result of batch 2:
## INFO [08:15:46.419] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:15:46.419] [bbotk] 2 imputemean 0.27902 0 0 1.40
## INFO [08:15:46.419] [bbotk] 2 imputerpart 0.28302 0 0 2.42
## INFO [08:15:46.419] [bbotk] 7 imputesample 0.27103 0 0 2.28
## INFO [08:15:46.419] [bbotk] 20 imputesample 0.24700 0 0 1.93
## INFO [08:15:46.419] [bbotk] uhash
## INFO [08:15:46.419] [bbotk] 638ead8f-0df1-4f6f-a22b-0bb557e4b40d
## INFO [08:15:46.419] [bbotk] ef23dd95-596e-405f-9d40-7e88dcecaff7
## INFO [08:15:46.419] [bbotk] 22eeb3ca-1dd8-4cfb-aeb6-5e25a6496d30
## INFO [08:15:46.419] [bbotk] 16904238-14c7-46d6-afe1-4d6c95e1b026
## INFO [08:15:46.425] [bbotk] Evaluating 4 configuration(s)
## INFO [08:15:52.599] [bbotk] Result of batch 3:
## INFO [08:15:52.607] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:15:52.607] [bbotk] 2 imputesample 0.27902 0 0 1.34
## INFO [08:15:52.607] [bbotk] 4 imputemean 0.27503 0 0 1.51
## INFO [08:15:52.607] [bbotk] 12 imputemean 0.27500 0 0 1.35
## INFO [08:15:52.607] [bbotk] 18 imputemean 0.26800 0 0 1.36
## INFO [08:15:52.607] [bbotk] uhash
## INFO [08:15:52.607] [bbotk] 3341b637-cc98-432b-ba88-322c8de52e6c
## INFO [08:15:52.607] [bbotk] 93d74162-d537-4386-ae5f-bfa8ddd9d1a7
## INFO [08:15:52.607] [bbotk] facb6361-c836-4fb4-84ec-9e573e39f56b
## INFO [08:15:52.607] [bbotk] 10da7fa2-6d22-48a0-a26d-47e2cca7f080
## INFO [08:15:52.615] [bbotk] Evaluating 4 configuration(s)
## INFO [08:16:02.178] [bbotk] Result of batch 4:
## INFO [08:16:02.183] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:16:02.183] [bbotk] 4 imputesample 0.29103 0 0 1.83
## INFO [08:16:02.183] [bbotk] 12 imputesample 0.26200 0 0 2.27
## INFO [08:16:02.183] [bbotk] 18 imputerpart 0.26399 0 0 2.34
## INFO [08:16:02.183] [bbotk] 20 imputerpart 0.24699 0 0 2.58
## INFO [08:16:02.183] [bbotk] uhash
## INFO [08:16:02.183] [bbotk] 4c93b1e8-d0ac-44b8-addb-4e6405557d99
## INFO [08:16:02.183] [bbotk] 60ae3694-7f1c-4568-b474-9ce3c6175e6d
## INFO [08:16:02.183] [bbotk] 49ddcccb-a4a9-4623-99d9-f6603cf2c27d
## INFO [08:16:02.183] [bbotk] c676ad25-aa38-4ea0-9ec7-2fde327e1a6a
## INFO [08:16:02.189] [bbotk] Evaluating 4 configuration(s)
## INFO [08:16:12.186] [bbotk] Result of batch 5:
## INFO [08:16:12.194] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:16:12.194] [bbotk] 4 imputerpart 0.28302 0 0 2.44
## INFO [08:16:12.194] [bbotk] 7 imputerpart 0.26602 0 0 2.34
## INFO [08:16:12.194] [bbotk] 10 imputemean 0.28502 0 0 1.57
## INFO [08:16:12.194] [bbotk] 12 imputerpart 0.26501 0 0 2.91
## INFO [08:16:12.194] [bbotk] uhash
## INFO [08:16:12.194] [bbotk] cf867f9e-cb13-4599-b3bb-ce5402c14196
## INFO [08:16:12.194] [bbotk] 624c9d3b-c1b0-4d59-8988-631f0893676a
## INFO [08:16:12.194] [bbotk] 0bf01f5c-7dac-48a0-8d73-917ee6735005
## INFO [08:16:12.194] [bbotk] 61f18c47-a565-4d66-829f-1bdb0a82b51d
## INFO [08:16:12.204] [bbotk] Evaluating 4 configuration(s)
## INFO [08:16:21.060] [bbotk] Result of batch 6:
## INFO [08:16:21.067] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:16:21.067] [bbotk] 7 imputemean 0.26203 0 0 1.77
## INFO [08:16:21.067] [bbotk] 10 imputerpart 0.25502 0 0 2.68
## INFO [08:16:21.067] [bbotk] 15 imputemean 0.27499 0 0 1.37
## INFO [08:16:21.067] [bbotk] 15 imputerpart 0.27900 0 0 2.33
## INFO [08:16:21.067] [bbotk] uhash
## INFO [08:16:21.067] [bbotk] 1d4e7018-c41d-4ce9-a3eb-1472172cdf95
## INFO [08:16:21.067] [bbotk] ecbdd72b-270a-45cb-8849-dad0d6446a87
## INFO [08:16:21.067] [bbotk] 43f23def-a340-4dcf-bf7b-b221a3320e44
## INFO [08:16:21.067] [bbotk] 63ab7dac-3847-4c9e-9b77-4707c3fd3326
## INFO [08:16:21.106] [bbotk] Finished optimizing after 24 evaluation(s)
## INFO [08:16:21.109] [bbotk] Result:
## INFO [08:16:21.113] [bbotk] information_gain.filter.nfeat branch.selection learner_param_vals x_domain classif.ce
## INFO [08:16:21.113] [bbotk] <int> <char> <list> <list> <num>
## INFO [08:16:21.113] [bbotk] 20 imputerpart <list[5]> <list[2]> 0.24699Visualize Tuning Process
Visualize the tuning process using a ggplot for each of
the two tuned hyperparameters.
Hint 1:
Performance results of the 3-fold CV for each configuration could be
viewed via the achive$data field of the
AutoTuner. Use e.g. the ggplot() function to
analyze the relationship of the hyperparameter values and the
performance values classif.ce.
Hint 2:
Solution:
Click me:
library(ggplot2)
archive = glrn_tuned$archive$data
ggplot(archive, aes(x = branch.selection, y = classif.ce)) +
geom_boxplot()
ggplot(archive, aes(x = information_gain.filter.nfeat, y = classif.ce, col = branch.selection)) +
geom_line()
ggplot(archive,
aes(x = information_gain.filter.nfeat, y = branch.selection, fill = classif.ce)) +
geom_tile() +
scale_fill_viridis_c()
Extract the best HPs
Which of the hyperparameter combination was the best performing one?
Hint:
You can either inspect the plots in the previous exercise or you can
have a look on the $tuning_result field of the trained
AutoTuner.
Benchmark
Benchmark the previous AutoTuner (which automatically
sets the best hyperparameters of the ML pipeline) against a decision
tree (using its default hyperparameter values). Use 3-fold
cross-validation.
Hint:
Solution:
Click me:
future::plan("multicore", workers = 4L)
lrns = list(
glrn_tuned,
po("imputemean") %>>% ranger,
lrn("classif.rpart")
)
resampling = rsmp("cv", folds = 3)
design = benchmark_grid(list(task), lrns, resampling)
bmr = benchmark(design)
## INFO [08:16:24.492] [bbotk] Starting to optimize 2 parameter(s) with '<OptimizerBatchGridSearch>' and '<TerminatorNone>'
## INFO [08:16:24.522] [bbotk] Evaluating 4 configuration(s)
## INFO [08:16:32.775] [bbotk] Result of batch 1:
## INFO [08:16:32.782] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:16:32.782] [bbotk] 7 imputesample 0.27327 0 0 1.57
## INFO [08:16:32.782] [bbotk] 7 imputerpart 0.27027 0 0 2.19
## INFO [08:16:32.782] [bbotk] 18 imputerpart 0.23273 0 0 2.38
## INFO [08:16:32.782] [bbotk] 20 imputemean 0.23423 0 0 1.48
## INFO [08:16:32.782] [bbotk] uhash
## INFO [08:16:32.782] [bbotk] f89f0280-3d65-492d-9214-9b75074aa9eb
## INFO [08:16:32.782] [bbotk] 847b5f1e-026c-4d2a-a3ea-5d9ffa39fff0
## INFO [08:16:32.782] [bbotk] f6ff9905-280d-4366-b889-715d5c0d5e18
## INFO [08:16:32.782] [bbotk] 6b8b0568-a8c0-409f-9e70-915ebc3effeb
## INFO [08:16:32.791] [bbotk] Evaluating 4 configuration(s)
## INFO [08:16:39.528] [bbotk] Result of batch 2:
## INFO [08:16:39.541] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:16:39.541] [bbotk] 2 imputemean 0.27177 0 0 1.19
## INFO [08:16:39.541] [bbotk] 10 imputesample 0.27177 0 0 1.51
## INFO [08:16:39.541] [bbotk] 12 imputemean 0.26577 0 0 1.38
## INFO [08:16:39.541] [bbotk] 20 imputesample 0.24474 0 0 1.91
## INFO [08:16:39.541] [bbotk] uhash
## INFO [08:16:39.541] [bbotk] 44d7d972-d605-490c-bca6-d7f6bb383d29
## INFO [08:16:39.541] [bbotk] e711baac-60e6-4e7b-8717-75a2a0ab049b
## INFO [08:16:39.541] [bbotk] cdc08766-a7dc-4c96-a4a0-340487873b16
## INFO [08:16:39.541] [bbotk] c675cc75-4e15-4385-b3b8-846c0aadc22d
## INFO [08:16:39.554] [bbotk] Evaluating 4 configuration(s)
## INFO [08:16:47.341] [bbotk] Result of batch 3:
## INFO [08:16:47.349] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:16:47.349] [bbotk] 2 imputerpart 0.27928 0 0 2.06
## INFO [08:16:47.349] [bbotk] 15 imputemean 0.24174 0 0 1.41
## INFO [08:16:47.349] [bbotk] 18 imputesample 0.26126 0 0 1.58
## INFO [08:16:47.349] [bbotk] 20 imputerpart 0.24625 0 0 2.03
## INFO [08:16:47.349] [bbotk] uhash
## INFO [08:16:47.349] [bbotk] 71bf955b-6f01-442a-bc7f-7f847fb37338
## INFO [08:16:47.349] [bbotk] d2f9b597-6a63-408f-b0f8-b0a0714704b6
## INFO [08:16:47.349] [bbotk] 5f4d5377-9030-400c-a5e9-42c04cc26f1d
## INFO [08:16:47.349] [bbotk] 2549c663-9f55-4e42-be5a-0148fbc0f206
## INFO [08:16:47.357] [bbotk] Evaluating 4 configuration(s)
## INFO [08:16:54.741] [bbotk] Result of batch 4:
## INFO [08:16:54.746] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:16:54.746] [bbotk] 4 imputemean 0.27327 0 0 1.27
## INFO [08:16:54.746] [bbotk] 4 imputerpart 0.27177 0 0 2.07
## INFO [08:16:54.746] [bbotk] 7 imputemean 0.26426 0 0 1.35
## INFO [08:16:54.746] [bbotk] 15 imputesample 0.25225 0 0 2.02
## INFO [08:16:54.746] [bbotk] uhash
## INFO [08:16:54.746] [bbotk] 4ebe4f8e-8d70-486d-bd65-c780bd3822d4
## INFO [08:16:54.746] [bbotk] 623616cc-dea0-4fc0-b42a-3945c7a25129
## INFO [08:16:54.746] [bbotk] f47676c5-7048-4ce5-b9e7-9484cf777565
## INFO [08:16:54.746] [bbotk] cdbef568-8c6b-4a7d-aec3-c58f996970f8
## INFO [08:16:54.753] [bbotk] Evaluating 4 configuration(s)
## INFO [08:17:03.253] [bbotk] Result of batch 5:
## INFO [08:17:03.261] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:17:03.261] [bbotk] 10 imputerpart 0.26426 0 0 2.34
## INFO [08:17:03.261] [bbotk] 12 imputesample 0.24925 0 0 1.81
## INFO [08:17:03.261] [bbotk] 15 imputerpart 0.24024 0 0 2.22
## INFO [08:17:03.261] [bbotk] 18 imputemean 0.25075 0 0 1.43
## INFO [08:17:03.261] [bbotk] uhash
## INFO [08:17:03.261] [bbotk] fde5009f-725b-45dc-8d90-981a6050cdc6
## INFO [08:17:03.261] [bbotk] 133f79bd-6919-44ed-a0ad-26377bf80ecb
## INFO [08:17:03.261] [bbotk] adfdff15-b824-465a-96ab-9bffb6c42388
## INFO [08:17:03.261] [bbotk] 93e5ef0b-c73b-4aa1-8407-9ca2d26ff8f8
## INFO [08:17:03.271] [bbotk] Evaluating 4 configuration(s)
## INFO [08:17:10.487] [bbotk] Result of batch 6:
## INFO [08:17:10.492] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:17:10.492] [bbotk] 2 imputesample 0.27027 0 0 1.30
## INFO [08:17:10.492] [bbotk] 4 imputesample 0.28979 0 0 1.79
## INFO [08:17:10.492] [bbotk] 10 imputemean 0.26877 0 0 1.32
## INFO [08:17:10.492] [bbotk] 12 imputerpart 0.24625 0 0 2.13
## INFO [08:17:10.492] [bbotk] uhash
## INFO [08:17:10.492] [bbotk] e3c1a08d-f112-4447-9228-7a8b7eb942aa
## INFO [08:17:10.492] [bbotk] f559e237-981d-42e9-974c-8a73c78074bb
## INFO [08:17:10.492] [bbotk] 1391ae67-95b3-48a6-aa1a-1350394d881f
## INFO [08:17:10.492] [bbotk] b5232681-f007-4134-9e83-2b84200fda55
## INFO [08:17:10.519] [bbotk] Finished optimizing after 24 evaluation(s)
## INFO [08:17:10.521] [bbotk] Result:
## INFO [08:17:10.525] [bbotk] information_gain.filter.nfeat branch.selection learner_param_vals x_domain classif.ce
## INFO [08:17:10.525] [bbotk] <int> <char> <list> <list> <num>
## INFO [08:17:10.525] [bbotk] 18 imputerpart <list[5]> <list[2]> 0.23273
## INFO [08:17:11.756] [bbotk] Starting to optimize 2 parameter(s) with '<OptimizerBatchGridSearch>' and '<TerminatorNone>'
## INFO [08:17:11.809] [bbotk] Evaluating 4 configuration(s)
## INFO [08:17:19.116] [bbotk] Result of batch 1:
## INFO [08:17:19.127] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:17:19.127] [bbotk] 4 imputemean 0.29833 0 0 1.27
## INFO [08:17:19.127] [bbotk] 12 imputemean 0.25781 0 0 1.18
## INFO [08:17:19.127] [bbotk] 18 imputerpart 0.25483 0 0 2.30
## INFO [08:17:19.127] [bbotk] 20 imputemean 0.23835 0 0 1.77
## INFO [08:17:19.127] [bbotk] uhash
## INFO [08:17:19.127] [bbotk] bc45922d-b526-451e-8f8b-4eade7ed5ce6
## INFO [08:17:19.127] [bbotk] 25342b1c-a8cd-4017-b198-56a326942597
## INFO [08:17:19.127] [bbotk] 69356c35-fc57-4000-a856-13467bc713d1
## INFO [08:17:19.127] [bbotk] cc48913f-a675-4f57-b48d-b38635551c3b
## INFO [08:17:19.139] [bbotk] Evaluating 4 configuration(s)
## INFO [08:17:26.244] [bbotk] Result of batch 2:
## INFO [08:17:26.250] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:17:26.250] [bbotk] 2 imputemean 0.27883 0 0 1.04
## INFO [08:17:26.250] [bbotk] 4 imputerpart 0.29234 0 0 2.14
## INFO [08:17:26.250] [bbotk] 18 imputemean 0.25638 0 0 1.35
## INFO [08:17:26.250] [bbotk] 20 imputesample 0.25636 0 0 1.83
## INFO [08:17:26.250] [bbotk] uhash
## INFO [08:17:26.250] [bbotk] 7a59a85c-96e2-4319-9798-d1c68082be7c
## INFO [08:17:26.250] [bbotk] 0bacc22a-4087-4fc9-99c1-fa2c9529b510
## INFO [08:17:26.250] [bbotk] f58ff89a-232f-4c7c-ba16-40bf21203f6e
## INFO [08:17:26.250] [bbotk] b32db0a5-dd10-4e4f-a157-d7c63b84991e
## INFO [08:17:26.256] [bbotk] Evaluating 4 configuration(s)
## INFO [08:17:34.360] [bbotk] Result of batch 3:
## INFO [08:17:34.365] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:17:34.365] [bbotk] 2 imputesample 0.27883 0 0 1.52
## INFO [08:17:34.365] [bbotk] 4 imputesample 0.29533 0 0 1.87
## INFO [08:17:34.365] [bbotk] 12 imputesample 0.27131 0 0 1.75
## INFO [08:17:34.365] [bbotk] 12 imputerpart 0.25484 0 0 2.29
## INFO [08:17:34.365] [bbotk] uhash
## INFO [08:17:34.365] [bbotk] d2a1c630-34ed-4d07-811e-6b86e9b37c14
## INFO [08:17:34.365] [bbotk] 7efd70dc-b6c2-4992-93c5-4cbf26bca1bf
## INFO [08:17:34.365] [bbotk] d002ffac-33c0-4653-888e-e00bfd52ae86
## INFO [08:17:34.365] [bbotk] 321a3c60-141f-4329-a3cf-63a2556f789a
## INFO [08:17:34.372] [bbotk] Evaluating 4 configuration(s)
## INFO [08:17:42.888] [bbotk] Result of batch 4:
## INFO [08:17:42.893] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:17:42.893] [bbotk] 2 imputerpart 0.27734 0 0 2.46
## INFO [08:17:42.893] [bbotk] 7 imputesample 0.26686 0 0 1.88
## INFO [08:17:42.893] [bbotk] 10 imputerpart 0.26385 0 0 2.26
## INFO [08:17:42.893] [bbotk] 15 imputemean 0.25488 0 0 1.20
## INFO [08:17:42.893] [bbotk] uhash
## INFO [08:17:42.893] [bbotk] b5ae21e9-230e-4d90-b00a-c0927c087bd9
## INFO [08:17:42.893] [bbotk] 9abfb230-0765-4cff-b42d-8d702e743faa
## INFO [08:17:42.893] [bbotk] 1c13ed3c-f645-4907-83d6-00c48dfc3636
## INFO [08:17:42.893] [bbotk] 4e99bc27-dd34-4841-a7e4-03634f19cffe
## INFO [08:17:42.900] [bbotk] Evaluating 4 configuration(s)
## INFO [08:17:52.243] [bbotk] Result of batch 5:
## INFO [08:17:52.248] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:17:52.248] [bbotk] 7 imputemean 0.27735 0 0 1.55
## INFO [08:17:52.248] [bbotk] 7 imputerpart 0.27584 0 0 2.38
## INFO [08:17:52.248] [bbotk] 15 imputerpart 0.24286 0 0 2.71
## INFO [08:17:52.248] [bbotk] 18 imputesample 0.24889 0 0 2.18
## INFO [08:17:52.248] [bbotk] uhash
## INFO [08:17:52.248] [bbotk] 70b0ffd5-d546-426d-a37e-28c42cbb520a
## INFO [08:17:52.248] [bbotk] 07940127-7e12-4d56-a75a-eba6281d7808
## INFO [08:17:52.248] [bbotk] 6a0dce32-d033-4951-90e4-06b358294986
## INFO [08:17:52.248] [bbotk] fd4f72da-e401-4ae6-82da-f0a824efde69
## INFO [08:17:52.253] [bbotk] Evaluating 4 configuration(s)
## INFO [08:17:59.231] [bbotk] Result of batch 6:
## INFO [08:17:59.240] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:17:59.240] [bbotk] 10 imputemean 0.26686 0 0 1.05
## INFO [08:17:59.240] [bbotk] 10 imputesample 0.26986 0 0 1.73
## INFO [08:17:59.240] [bbotk] 15 imputesample 0.27135 0 0 1.52
## INFO [08:17:59.240] [bbotk] 20 imputerpart 0.23382 0 0 2.13
## INFO [08:17:59.240] [bbotk] uhash
## INFO [08:17:59.240] [bbotk] 40ca71aa-203c-40d5-b21c-514036fe92c6
## INFO [08:17:59.240] [bbotk] 014196db-7d30-4168-8536-7ce908f77d79
## INFO [08:17:59.240] [bbotk] aadf68fb-84ce-4175-bfb1-5914b07d42a6
## INFO [08:17:59.240] [bbotk] 8d52b1fa-6f35-443f-8544-12f05349c2b1
## INFO [08:17:59.291] [bbotk] Finished optimizing after 24 evaluation(s)
## INFO [08:17:59.295] [bbotk] Result:
## INFO [08:17:59.305] [bbotk] information_gain.filter.nfeat branch.selection learner_param_vals x_domain classif.ce
## INFO [08:17:59.305] [bbotk] <int> <char> <list> <list> <num>
## INFO [08:17:59.305] [bbotk] 20 imputerpart <list[5]> <list[2]> 0.23382
## INFO [08:18:00.555] [bbotk] Starting to optimize 2 parameter(s) with '<OptimizerBatchGridSearch>' and '<TerminatorNone>'
## INFO [08:18:00.580] [bbotk] Evaluating 4 configuration(s)
## INFO [08:18:06.595] [bbotk] Result of batch 1:
## INFO [08:18:06.601] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:18:06.601] [bbotk] 4 imputemean 0.29685 0 0 1.17
## INFO [08:18:06.601] [bbotk] 4 imputesample 0.28934 0 0 1.44
## INFO [08:18:06.601] [bbotk] 12 imputemean 0.27438 0 0 1.20
## INFO [08:18:06.601] [bbotk] 18 imputemean 0.27586 0 0 1.51
## INFO [08:18:06.601] [bbotk] uhash
## INFO [08:18:06.601] [bbotk] 2ee43ace-0e0d-4a5f-b0aa-384961ac437c
## INFO [08:18:06.601] [bbotk] 61bea917-0499-4a37-840b-36c39e52b4a5
## INFO [08:18:06.601] [bbotk] 6c6cb532-f633-4dd8-9218-489c9e6358ee
## INFO [08:18:06.601] [bbotk] e8c3f4c1-eaf1-4980-9203-a2750a59f798
## INFO [08:18:06.609] [bbotk] Evaluating 4 configuration(s)
## INFO [08:18:14.643] [bbotk] Result of batch 2:
## INFO [08:18:14.649] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:18:14.649] [bbotk] 2 imputesample 0.32835 0 0 1.55
## INFO [08:18:14.649] [bbotk] 10 imputesample 0.25938 0 0 1.69
## INFO [08:18:14.649] [bbotk] 12 imputesample 0.26989 0 0 1.67
## INFO [08:18:14.649] [bbotk] 12 imputerpart 0.26688 0 0 2.39
## INFO [08:18:14.649] [bbotk] uhash
## INFO [08:18:14.649] [bbotk] cab12319-34d7-4334-ae5a-cb6e4f6f4201
## INFO [08:18:14.649] [bbotk] 8d091be7-70f4-47e3-ae02-5b7659f4d8c3
## INFO [08:18:14.649] [bbotk] bdfecb0e-3061-4693-b84d-dbb25f0f29cb
## INFO [08:18:14.649] [bbotk] e4cba4f4-0530-418b-bec9-4ca587d8b6b3
## INFO [08:18:14.657] [bbotk] Evaluating 4 configuration(s)
## INFO [08:18:21.201] [bbotk] Result of batch 3:
## INFO [08:18:21.208] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:18:21.208] [bbotk] 2 imputemean 0.31785 0 0 1.24
## INFO [08:18:21.208] [bbotk] 4 imputerpart 0.28484 0 0 1.91
## INFO [08:18:21.208] [bbotk] 15 imputerpart 0.25187 0 0 1.66
## INFO [08:18:21.208] [bbotk] 20 imputesample 0.25938 0 0 1.17
## INFO [08:18:21.208] [bbotk] uhash
## INFO [08:18:21.208] [bbotk] d8dc1621-756d-430b-8521-217d58857d37
## INFO [08:18:21.208] [bbotk] 2de2a199-6b11-4310-ab3d-6019a8af56c7
## INFO [08:18:21.208] [bbotk] a5e52fe9-b606-470c-b2ba-0935ec6b0bac
## INFO [08:18:21.208] [bbotk] adff41a6-da59-496b-885f-fe9efbd6da23
## INFO [08:18:21.219] [bbotk] Evaluating 4 configuration(s)
## INFO [08:18:28.182] [bbotk] Result of batch 4:
## INFO [08:18:28.187] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:18:28.187] [bbotk] 7 imputesample 0.27586 0 0 1.07
## INFO [08:18:28.187] [bbotk] 18 imputesample 0.25938 0 0 1.45
## INFO [08:18:28.187] [bbotk] 18 imputerpart 0.24893 0 0 2.50
## INFO [08:18:28.187] [bbotk] 20 imputemean 0.25487 0 0 1.39
## INFO [08:18:28.187] [bbotk] uhash
## INFO [08:18:28.187] [bbotk] 236cae09-cb47-454f-a62e-5e4e4c7a7bdb
## INFO [08:18:28.187] [bbotk] c087dd53-88ef-48ee-bba8-0a18382c82f6
## INFO [08:18:28.187] [bbotk] 3ca047b3-4d53-4e37-80e5-556a10cc33e1
## INFO [08:18:28.187] [bbotk] 179a65f4-11e8-485e-a133-d1f930070030
## INFO [08:18:28.193] [bbotk] Evaluating 4 configuration(s)
## INFO [08:18:36.016] [bbotk] Result of batch 5:
## INFO [08:18:36.023] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:18:36.023] [bbotk] 7 imputemean 0.26686 0 0 1.19
## INFO [08:18:36.023] [bbotk] 7 imputerpart 0.27285 0 0 1.94
## INFO [08:18:36.023] [bbotk] 10 imputerpart 0.25938 0 0 2.30
## INFO [08:18:36.023] [bbotk] 15 imputesample 0.25488 0 0 1.77
## INFO [08:18:36.023] [bbotk] uhash
## INFO [08:18:36.023] [bbotk] ee586e8f-aaa7-4633-aef2-fda8f13135e8
## INFO [08:18:36.023] [bbotk] 5adb65d9-9757-4a14-8c85-09ea822bc692
## INFO [08:18:36.023] [bbotk] 195f09f9-18a4-4441-ae4b-d4330a5f84ed
## INFO [08:18:36.023] [bbotk] dbc94fa9-b985-4d33-bfe6-c68da2fbf213
## INFO [08:18:36.029] [bbotk] Evaluating 4 configuration(s)
## INFO [08:18:44.383] [bbotk] Result of batch 6:
## INFO [08:18:44.387] [bbotk] information_gain.filter.nfeat branch.selection classif.ce warnings errors runtime_learners
## INFO [08:18:44.387] [bbotk] 2 imputerpart 0.33132 0 0 2.00
## INFO [08:18:44.387] [bbotk] 10 imputemean 0.25787 0 0 1.55
## INFO [08:18:44.387] [bbotk] 15 imputemean 0.25489 0 0 1.67
## INFO [08:18:44.387] [bbotk] 20 imputerpart 0.23990 0 0 2.50
## INFO [08:18:44.387] [bbotk] uhash
## INFO [08:18:44.387] [bbotk] fe8eff92-b5a3-476a-a0ad-35240f65af8a
## INFO [08:18:44.387] [bbotk] f5b90e16-ca58-437a-bfc1-73ffb235b051
## INFO [08:18:44.387] [bbotk] 7812537a-f4dc-47df-a60a-c51faa47498c
## INFO [08:18:44.387] [bbotk] 2a52284f-99ad-40c8-90d5-0f170b660efb
## INFO [08:18:44.412] [bbotk] Finished optimizing after 24 evaluation(s)
## INFO [08:18:44.414] [bbotk] Result:
## INFO [08:18:44.417] [bbotk] information_gain.filter.nfeat branch.selection learner_param_vals x_domain classif.ce
## INFO [08:18:44.417] [bbotk] <int> <char> <list> <list> <num>
## INFO [08:18:44.417] [bbotk] 20 imputerpart <list[5]> <list[2]> 0.2399
bmr$aggregate()
## nr task_id
## <int> <char>
## 1: 1 german_credit_NA
## 2: 2 german_credit_NA
## 3: 3 german_credit_NA
## learner_id resampling_id
## <char> <char>
## 1: branch.imputemean.imputesample.imputelearner.unbranch.information_gain.classif.ranger.tuned cv
## 2: imputemean.classif.ranger cv
## 3: classif.rpart cv
## iters classif.ce
## <int> <num>
## 1: 3 0.23197
## 2: 3 0.23797
## 3: 3 0.27598
## Hidden columns: resample_result
autoplot(bmr)
Summary
In this exercise sheet, we learned how to tune the whole pipeline
such that preprocessing as well as model fitting can be optimized for
the task at hand. We set up an AutoTuner object that
combines the GraphLearner with the Tuner and
could be used as a proper mlr3 learner. We compared the
AutoTuner with some other learners.
Of course, we only saw a selection of the full functionality of the
mlr3pipelines package - if you want to learn more have a
look in the mlr3book.