library(dplyr) # for data manipulation
library(ggplot2) # for awesome graphics
library(rsample) # splitting data
library(caret) # for cross-validation, etc.
library(vip) # variable importance
library(modeldata) # ames data
data(ames)
# Stratified sampling with the rsample package
set.seed(77654) # I used a different seed than in the book
split <- initial_split(ames, prop = 0.7, strata = "Sale_Price")
ames_train <- training(split)
ames_test <- testing(split)
mars1 <- earth::earth(
Sale_Price ~ .,
data = ames_train
)Boook club “Hands-On Machine Learning with R” #5
Chp7 Splines - Chp 8 KNN
Capturing non-linear relationships
Polynomials
Step functions
Require specifications by the user
- Which variables should have what specific degree of interaction or at what points of a variable \(X\) should cut points be made for the step functions
Multivariate adaptive regression splines (MARS)
Capture the nonlinear relationships in the data by assessing cutpoints (knots) similar to step functions
The procedure assesses each data point for each predictor as a knot and creates a linear regression model with the candidate feature(s)
Many knots may have a good fit in training data, but may not generalize to new data
Pruning: remove knots that do not contribute to predictive accuracy using, e.g. cross-validation
Performance and residual plots
- Generalized cross-validation (GCV) \(R^2\) (solid black line; left-hand y-axis)
- Number of terms retained in the model (x-axis)
- Number of original predictors (right-hand y-axis)
Tuning hyperparameters with caret
- Grid search using 10-fold CV
- The optimal model’s cross-validated RMSE was $26,817 in the book and $27,246.61 in this example with different seed
- The optimal model retains 56 terms and includes up to 2nd degree interactions in the book and 45 terms and 1 degree interactions with the seed I chose
RMSE Rsquared MAE Resample
Min. :21382 Min. :0.8429 Min. :15338 Length:10
1st Qu.:25083 1st Qu.:0.8684 1st Qu.:16760 Class :character
Median :27005 Median :0.8781 Median :18240 Mode :character
Mean :27247 Mean :0.8845 Mean :17822
3rd Qu.:29091 3rd Qu.:0.9046 3rd Qu.:18775
Max. :35227 Max. :0.9242 Max. :19859 Comparing MARS with other modelling approaches
RMSE Rsquared MAE Resample
1 26592.75 0.8747269 18206.57 Fold06
2 28382.27 0.8990683 18886.13 Fold03
3 21381.58 0.9165223 15338.32 Fold07
4 25412.92 0.8811599 16541.65 Fold10
5 35226.86 0.8428650 19858.67 Fold04
6 29327.27 0.8663435 18838.37 Fold08
7 27417.38 0.8749882 18584.27 Fold01
8 23167.69 0.9241581 16278.33 Fold05
9 30584.40 0.8584081 18273.82 Fold09
10 24972.94 0.9064159 17415.43 Fold02
Feature interpretation
earthhas backwards elimination feature selection tool- This tool looks at reductions in the GCV estimate of error as each predictor is added to the model:
value = "gcv" - MARS automatically includes and excludes terms during the pruning process (automated feature selection)
- Feature never included in the final model-> importance value=0
- An alternative: the change in the residual sums of squares (RSS) as terms are added (
value = "rss") - No measuring of the impact for particular hinge functions created for a given feature
Feature interpretation: hinge functions interactions
- Investigate interactions
- Create partial dependence plots (PDPs) for each feature individually and also together
- Model found that one knot in each feature provides the best fit
- Gr_Liv_Area increases and for newer homes, Sale_Price increases dramatically
# Construct partial dependence plots
p1 <- partial(cv_mars, pred.var = "Gr_Liv_Area", grid.resolution = 10) %>%
autoplot()
p2 <- partial(cv_mars, pred.var = "Year_Built", grid.resolution = 10) %>%
autoplot()
p3 <- partial(cv_mars, pred.var = c("Gr_Liv_Area", "Year_Built"),
grid.resolution = 10) %>%
plotPartial(levelplot = FALSE, zlab = "yhat", drape = TRUE, colorkey = TRUE,
screen = list(z = -20, x = -60))
# Display plots side by side
gridExtra::grid.arrange(p1, p2, p3, ncol = 3)