10 Second wave

In this section we move to the second wave of emulation. We will start by defining all the data necessary to train the second wave of emulators. We will then go through the same steps as in the previous section to train the emulators, test them and generate new points.

To perform a second wave of history matching and emulation we follow the same procedure as in the previous sections, with two caveats. We start by forming a dataframe wave1 using parameters sets in new_points, as we did with wave0, i.e. we run the model at new_points using the function get_results and then bind the obtained outputs to new_points. Half of wave1 should be used as the training set for the new emulators, and the other half as the validation set to evaluate the new emulators’ performance.

Now note that parameter sets in new_points tend to lie in a small region inside the original input space, since new_points contains only non-implausible points, according to the first wave emulators. The first caveat is then that it is preferable to train the new emulators only on the non-implausible region found in wave one. This can be done by setting the argument check.ranges to TRUE in the function emulator_from_data. The second caveat is that emulators from both the first and second wave need to be passed to the generate_new_design function: this is because second wave emulators tend to be rather uncertain outside of the small region of the input space where they were trained.

In the task below, you can have a go at wave 2 of the emulation and history matching process yourself.

Task

Using new_points and setting check.ranges to TRUE, train new emulators. Customise them and generate new parameter sets.

Solution

We start by evaluating the function get_results on new_points. In other words, we run the model using the parameter sets we generated at the end of wave 1:

new_initial_results <- setNames(data.frame(t(apply(new_points, 1, get_results, 
                                c(25, 40, 100, 200, 300, 350), c('I', 'R')))), 
                                names(targets))

and then binding new_points to the model runs new_initial_results to create the data.frame wave1, containing the input parameter sets and model outputs:

wave1 <- cbind(new_points, new_initial_results)

We split wave1 into training and validation sets

new_t_sample <- sample(1:nrow(wave1), 90)
new_training <- wave1[new_t_sample,]
new_validation <- wave1[-new_t_sample,]

and train wave two emulators on the space defined by new_ranges:

ems_wave2 <- emulator_from_data(new_training, names(targets), ranges, 
                                check.ranges=TRUE,
                                c_lengths= rep(0.55,length(targets)))

## I25 
## I40 
## I100 
## I200 
## I300 
## I350 
## R25 
## R40 
## R100 
## R200 
## R300 
## R350 
## I25 
## I40 
## I100 
## I200 
## I300 
## I350 
## R25 
## R40 
## R100 
## R200 
## R300 
## R350

Let us check their diagnostics:

vd <- validation_diagnostics(ems_wave2, validation = new_validation, targets = targets, 
                             plt=TRUE)

Most emulators fail at least one of the three dagnostics. Let us modify the sigmas in order to build more conservative emulators that pass the three diagnostics. After some trial and error, we chose the following values of sigma for our emulators:

inflations <- c(2,4,2,2,2,1,3,2,2,2,2,2)
for (i in 1:length(ems_wave2)) {
ems_wave2[[i]] <- ems_wave2[[i]]$mult_sigma(inflations[[i]])
}
vd <- validation_diagnostics(ems_wave2, validation =  new_validation, 
                             targets = targets, plt=TRUE)

The diagnostics look ok now. Let us try to generate new parameter sets using all emulators built so far:

new_new_points <- generate_new_design(c(ems_wave2, ems_wave1), 180, targets, verbose=TRUE)

## Proposing from LHS...
## LHS has high yield; no other methods required.
## Selecting final points using maximin criterion...

plot_wrap(new_new_points, ranges)

This worked well: the new non-implausible region is clearly smaller than the one we had at the end of wave one.