3.1.1 Delete some rows

Since gross and budget have too many missing values, and we want to keep these two variables for the following analysis, we can only delete rows with null values for gross and budget because imputation will not do a good job here.

IMDB <- IMDB[!is.na(IMDB$gross), ]
IMDB <- IMDB[!is.na(IMDB$budget), ]
dim(IMDB)

## [1] 3857   27

Not too bad, we only omitted 23% of the observations. Now our data has 3857 observations.

Let’s see how many complete cases we have.

sum(complete.cases(IMDB))

## [1] 3768

So, there are still 3857 – 3768 = 89 rows with NAs.

3.1.2 Analyze aspect ratio

let’s take a look at rest columns with missing values.

colSums(sapply(IMDB, is.na))

##                     color             director_name 
##                         0                         0 
##    num_critic_for_reviews                  duration 
##                         1                         1 
##   director_facebook_likes    actor_3_facebook_likes 
##                         0                        10 
##              actor_2_name    actor_1_facebook_likes 
##                         0                         3 
##                     gross              actor_1_name 
##                         0                         0 
##               movie_title           num_voted_users 
##                         0                         0 
## cast_total_facebook_likes              actor_3_name 
##                         0                         0 
##      facenumber_in_poster             plot_keywords 
##                         6                         0 
##           movie_imdb_link      num_user_for_reviews 
##                         0                         0 
##                  language                   country 
##                         0                         0 
##            content_rating                    budget 
##                         0                         0 
##                title_year    actor_2_facebook_likes 
##                         0                         5 
##                imdb_score              aspect_ratio 
##                         0                        74 
##      movie_facebook_likes 
##                         0

Now aspect_ratio has the highest number of missing values. Before trying to impute the missing values, we want to check how important is this variable.

table(IMDB$aspect_ratio)

## 
## 1.18 1.33 1.37  1.5 1.66 1.75 1.77 1.78 1.85    2  2.2 2.24 2.35 2.39  2.4 
##    1   19   50    1   40    2    1   41 1600    3   10    1 1995   11    3 
## 2.55 2.76   16 
##    1    3    1

The most common aspect ratios are 1.85 and 2.35. For analyzing purpose, we group other ratios together.

In order to compute the mean of imdb score for different aspect_ratio, we need to replace NA with 0 first.

IMDB$aspect_ratio[is.na(IMDB$aspect_ratio)] <- 0
mean(IMDB$imdb_score[IMDB$aspect_ratio == 1.85])

## [1] 6.373938

mean(IMDB$imdb_score[IMDB$aspect_ratio == 2.35])

## [1] 6.508471

mean(IMDB$imdb_score[IMDB$aspect_ratio != 1.85 & IMDB$aspect_ratio != 2.35])

## [1] 6.672519

From the means of imdb score for different aspect ratios, we can see there is no significant difference, all the means fall in the range of 6.3~6.8. So, removing this variable won’t affect our following analysis.

IMDB <- subset(IMDB, select = -c(aspect_ratio))

3.1.3 Deal with 0s

We notice that there are some 0 values which should also be regarded as missing value except for predictor facenumber_in_poster.

First we need to replace NA with column average for facenumber_in_poster, then replace 0s in other predictors with NA, and lastly replace all NAs with their respective column mean.

# replace NA with column average for facenumber_in_poster
IMDB$facenumber_in_poster[is.na(IMDB$facenumber_in_poster)] <- round(mean(IMDB$facenumber_in_poster, na.rm = TRUE))
# convert 0s into NAs for other predictors
IMDB[,c(5,6,8,13,24,26)][IMDB[,c(5,6,8,13,24,26)] == 0] <- NA
# impute missing value with column mean
IMDB$num_critic_for_reviews[is.na(IMDB$num_critic_for_reviews)] <- round(mean(IMDB$num_critic_for_reviews, na.rm = TRUE))
IMDB$duration[is.na(IMDB$duration)] <- round(mean(IMDB$duration, na.rm = TRUE))
IMDB$director_facebook_likes[is.na(IMDB$director_facebook_likes)] <- round(mean(IMDB$director_facebook_likes, na.rm = TRUE))
IMDB$actor_3_facebook_likes[is.na(IMDB$actor_3_facebook_likes)] <- round(mean(IMDB$actor_3_facebook_likes, na.rm = TRUE))
IMDB$actor_1_facebook_likes[is.na(IMDB$actor_1_facebook_likes)] <- round(mean(IMDB$actor_1_facebook_likes, na.rm = TRUE))
IMDB$cast_total_facebook_likes[is.na(IMDB$cast_total_facebook_likes)] <- round(mean(IMDB$cast_total_facebook_likes, na.rm = TRUE))
IMDB$actor_2_facebook_likes[is.na(IMDB$actor_2_facebook_likes)] <- round(mean(IMDB$actor_2_facebook_likes, na.rm = TRUE))
IMDB$movie_facebook_likes[is.na(IMDB$movie_facebook_likes)] <- round(mean(IMDB$movie_facebook_likes, na.rm = TRUE))

Now we finished imputing the numeric missing values. There are still some categorical missing values, let’s take a look.

3.1.4 Sort out content ratings

We find there are still some missing values in content_rating, which are marked as “”.

table(IMDB$content_rating)

## 
##            Approved         G        GP         M     NC-17 Not Rated 
##        51        17        91         1         2         6        42 
##    Passed        PG     PG-13         R     TV-14      TV-G     TV-MA 
##         3       573      1314      1723         0         0         0 
##     TV-PG      TV-Y     TV-Y7   Unrated         X 
##         0         0         0        24        10

Blanks should be taken as missing value. Since these missing values cannot be replaced with reasonable data, we delete these rows.

IMDB <- IMDB[!(IMDB$content_rating %in% ""),]

According to the history of naming these different content ratings, we find M = GP = PG, X = NC-17. We want to replace M and GP with PG, replace X with NC-17, because these two are what we use nowadays.

IMDB$content_rating[IMDB$content_rating == 'M']   <- 'PG' 
IMDB$content_rating[IMDB$content_rating == 'GP']  <- 'PG' 
IMDB$content_rating[IMDB$content_rating == 'X']   <- 'NC-17'

We want to replace “Approved”, “Not Rated”, “Passed”, “Unrated” with the most common rating “R”.

IMDB$content_rating[IMDB$content_rating == 'Approved']  <- 'R' 
IMDB$content_rating[IMDB$content_rating == 'Not Rated'] <- 'R' 
IMDB$content_rating[IMDB$content_rating == 'Passed']    <- 'R' 
IMDB$content_rating[IMDB$content_rating == 'Unrated']   <- 'R' 
IMDB$content_rating <- factor(IMDB$content_rating)
table(IMDB$content_rating)

## 
##     G NC-17    PG PG-13     R 
##    91    16   576  1314  1809

Now we only have 5 different content ratings.

3.3.1 Is the color of a movie influential?

table(IMDB$color)

## 
##                   Black and White            Color 
##                2              124             3680

More than 96% movies are colored, which indicates that this predictor is nearly constant. Let’s remove this predictor.

# delete predictor color
IMDB <- subset(IMDB, select = -c(color))

3.3.2 Is language an important factor for imdb score? What about country?

table(IMDB$language)

## 
##            Aboriginal     Arabic    Aramaic    Bosnian  Cantonese 
##          2          2          1          1          1          7 
##    Chinese      Czech     Danish       Dari      Dutch   Dzongkha 
##          0          1          3          2          3          0 
##    English   Filipino     French     German      Greek     Hebrew 
##       3644          1         34         11          0          2 
##      Hindi  Hungarian  Icelandic Indonesian    Italian   Japanese 
##          5          1          0          2          7         10 
##    Kannada     Kazakh     Korean   Mandarin       Maya  Mongolian 
##          0          1          5         14          1          1 
##       None  Norwegian    Panjabi    Persian     Polish Portuguese 
##          1          4          0          3          0          5 
##   Romanian    Russian  Slovenian    Spanish    Swahili    Swedish 
##          1          1          0         24          0          0 
##      Tamil     Telugu       Thai       Urdu Vietnamese       Zulu 
##          0          0          3          0          1          1

Over 95% movies are in English, which means this variable is nearly constant. Let’s remove it.

IMDB <- subset(IMDB, select = -c(language))

Let’s take a look at predictor country.

table(IMDB$country)

## 
##                               Afghanistan            Argentina 
##                    0                    1                    3 
##                Aruba            Australia              Bahamas 
##                    1                   40                    0 
##              Belgium               Brazil             Bulgaria 
##                    1                    5                    0 
##             Cambodia             Cameroon               Canada 
##                    0                    0                   63 
##                Chile                China             Colombia 
##                    1                   13                    1 
##       Czech Republic              Denmark   Dominican Republic 
##                    3                    9                    0 
##                Egypt              Finland               France 
##                    0                    1                  103 
##              Georgia              Germany               Greece 
##                    1                   79                    1 
##            Hong Kong              Hungary              Iceland 
##                   13                    2                    1 
##                India            Indonesia                 Iran 
##                    5                    1                    4 
##              Ireland               Israel                Italy 
##                    7                    2                   11 
##                Japan                Kenya           Kyrgyzstan 
##                   15                    0                    0 
##                Libya               Mexico          Netherlands 
##                    0                   10                    3 
##             New Line          New Zealand              Nigeria 
##                    1                   11                    0 
##               Norway        Official site             Pakistan 
##                    4                    1                    0 
##               Panama                 Peru          Philippines 
##                    0                    1                    1 
##               Poland              Romania               Russia 
##                    1                    2                    3 
##             Slovakia             Slovenia         South Africa 
##                    0                    0                    3 
##          South Korea         Soviet Union                Spain 
##                    8                    0                   22 
##               Sweden          Switzerland               Taiwan 
##                    0                    0                    2 
##             Thailand               Turkey                   UK 
##                    4                    0                  316 
## United Arab Emirates                  USA         West Germany 
##                    0                 3025                    1

Around 79% movies are from USA, 8% from UK, 13% from other countries. So we group other countries together to make this categorical variable with less levels: USA, UK, Others.

levels(IMDB$country) <- c(levels(IMDB$country), "Others")
IMDB$country[(IMDB$country != 'USA')&(IMDB$country != 'UK')] <- 'Others' 
IMDB$country <- factor(IMDB$country)
table(IMDB$country)

## 
##     UK    USA Others 
##    316   3025    465

6.1.1 Full-grown Tree

library(rpart)
library(rpart.plot)
# Full grown tree
class.tree <- rpart(binned_score ~ . -imdb_score, data = train, method = "class")
## plot tree
prp(class.tree, type = 1, extra = 1, under = TRUE, split.font = 2, varlen = 0) 

Classification rules:

• 1. If (user_vote >= 551000) then class = (8,10].
• 2. If (83000 <= user_vote < 551000) then class = (6,8].
• 3. If (user_vote < 83000) and (duration >= 106) then class = (6,8].
• 4. If (user_vote < 83000) and (duration < 106) and (gross < 7900000) then class = (6,8].
• 5. If (user_vote < 83000) and (duration < 106) and (gross >= 7900000) and (movie_fb < 4500) then class = (4,6].
• 6. If (user_vote < 83000) and (duration < 106) and (gross >= 7900000) and (movie_fb >= 4500) and (year < 2000) then class = (6,8].
• 7. If (user_vote < 83000) and (duration < 106) and (gross >= 7900000) and (movie_fb >= 4500) and (year >= 2000) and (critic_review_ratio < 1.2) then class = (4,6].
• 8. If (user_vote < 41000) and (duration < 106) and (gross >= 7900000) and (movie_fb >= 4500) and (year >= 2000) and (critic_review_ratio >= 1.2) then class = (4,6].
• 9. If (41000 <= user_vote < 83000) and (duration < 106) and (gross >= 7900000) and (movie_fb >= 4500) and (year >= 2000) and (critic_review_ratio >= 1.2) then class = (6,8].

From these rules, we can conclude that movies with a lot of votes in imdb website tend to have a higher score, which really makes sense because popular movies will have a lot of fans to vote high scores for them.

On the contrary, if a movie has fewer votes, it can still be a good movie if its duration is longer (rule #3).

It is surprise to see that movies make less profit are good, but ok if they make more profit (rule #4).

6.1.2 Best-pruned Tree

# cross-validation procedure
# argument cp sets the smallest value for the complexity parameter.
set.seed(51)
cv.ct <- rpart(binned_score ~ . -imdb_score, data = train, method = "class", 
               cp = 0.00001, minsplit = 5, xval = 5)
printcp(cv.ct)

## 
## Classification tree:
## rpart(formula = binned_score ~ . - imdb_score, data = train, 
##     method = "class", cp = 1e-05, minsplit = 5, xval = 5)
## 
## Variables actually used in tree construction:
##  [1] actor1_fb           budget              content            
##  [4] country             critic_review_ratio director_fb        
##  [7] duration            face_number         gross              
## [10] movie_fb            other_actors_fb     user_vote          
## [13] year               
## 
## Root node error: 798/2226 = 0.35849
## 
## n= 2226 
## 
##            CP nsplit rel error  xerror     xstd
## 1  0.06390977      0   1.00000 1.00000 0.028353
## 2  0.04636591      3   0.80827 0.86216 0.027322
## 3  0.01691729      4   0.76190 0.79574 0.026697
## 4  0.00751880      8   0.69424 0.77694 0.026503
## 5  0.00626566     10   0.67920 0.76316 0.026357
## 6  0.00563910     13   0.66040 0.75815 0.026303
## 7  0.00543024     15   0.64912 0.75815 0.026303
## 8  0.00501253     20   0.61278 0.75564 0.026276
## 9  0.00407268     25   0.58772 0.76817 0.026411
## 10 0.00375940     29   0.57143 0.77820 0.026517
## 11 0.00325815     45   0.50877 0.78070 0.026543
## 12 0.00322234     50   0.49248 0.78446 0.026582
## 13 0.00313283     57   0.46992 0.78446 0.026582
## 14 0.00292398     63   0.45113 0.79574 0.026697
## 15 0.00250627     66   0.44236 0.79574 0.026697
## 16 0.00219298    112   0.31830 0.80827 0.026821
## 17 0.00187970    120   0.29950 0.80451 0.026784
## 18 0.00167084    133   0.27444 0.80576 0.026797
## 19 0.00156642    142   0.25940 0.80576 0.026797
## 20 0.00125313    151   0.24436 0.83208 0.027049
## 21 0.00093985    200   0.18045 0.84712 0.027188
## 22 0.00083542    204   0.17669 0.84336 0.027154
## 23 0.00062657    207   0.17419 0.85714 0.027278
## 24 0.00050125    213   0.17043 0.85714 0.027278
## 25 0.00027847    218   0.16792 0.86842 0.027376
## 26 0.00025063    227   0.16541 0.86842 0.027376
## 27 0.00001000    237   0.16291 0.86842 0.027376

The 8th tree has the lowest cross-validation error (xerror): 0.75564.

# prune by lowest cp
pruned.ct <- prune(cv.ct, 
                   cp = cv.ct$cptable[which.min(cv.ct$cptable[,"xerror"]),"CP"])
length(pruned.ct$frame$var[pruned.ct$frame$var == "<leaf>"])

## [1] 21

prp(pruned.ct, type = 1, extra = 1, split.font = 1, varlen = -10)

6.1.3 Apply Model

# apply model on training set
tree.pred.train <- predict(pruned.ct, train, type = "class")
# generate confusion matrix for training data
confusionMatrix(tree.pred.train, train$binned_score)

## Confusion Matrix and Statistics
## 
##           Reference
## Prediction (0,4] (4,6] (6,8] (8,10]
##     (0,4]      0     0     0      0
##     (4,6]     45   378   115      0
##     (6,8]     22   266  1305     33
##     (8,10]     0     0     8     54
## 
## Overall Statistics
##                                           
##                Accuracy : 0.7803          
##                  95% CI : (0.7625, 0.7974)
##     No Information Rate : 0.6415          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.5228          
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: (0,4] Class: (4,6] Class: (6,8] Class: (8,10]
## Sensitivity                0.0000       0.5870       0.9139       0.62069
## Specificity                1.0000       0.8989       0.5977       0.99626
## Pos Pred Value                NaN       0.7026       0.8026       0.87097
## Neg Pred Value             0.9699       0.8424       0.7950       0.98475
## Prevalence                 0.0301       0.2893       0.6415       0.03908
## Detection Rate             0.0000       0.1698       0.5863       0.02426
## Detection Prevalence       0.0000       0.2417       0.7305       0.02785
## Balanced Accuracy          0.5000       0.7429       0.7558       0.80847

Accuracy is 0.7803 for training set.

# apply model on validation set
tree.pred.valid <- predict(pruned.ct, valid, type = "class")
# generate confusion matrix for validation data
confusionMatrix(tree.pred.valid, valid$binned_score)

## Confusion Matrix and Statistics
## 
##           Reference
## Prediction (0,4] (4,6] (6,8] (8,10]
##     (0,4]      0     0     0      0
##     (4,6]      9    88    62      0
##     (6,8]      6   121   424     10
##     (8,10]     0     0     5     17
## 
## Overall Statistics
##                                           
##                Accuracy : 0.7129          
##                  95% CI : (0.6789, 0.7453)
##     No Information Rate : 0.6617          
##     P-Value [Acc > NIR] : 0.001616        
##                                           
##                   Kappa : 0.345           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: (0,4] Class: (4,6] Class: (6,8] Class: (8,10]
## Sensitivity               0.00000       0.4211       0.8635       0.62963
## Specificity               1.00000       0.8668       0.4542       0.99301
## Pos Pred Value                NaN       0.5535       0.7558       0.77273
## Neg Pred Value            0.97978       0.7925       0.6298       0.98611
## Prevalence                0.02022       0.2817       0.6617       0.03639
## Detection Rate            0.00000       0.1186       0.5714       0.02291
## Detection Prevalence      0.00000       0.2143       0.7561       0.02965
## Balanced Accuracy         0.50000       0.6439       0.6589       0.81132

Accuracy is 0.7129 for validation set.

# apply model on test set
tree.pred.test <- predict(pruned.ct, test, type = "class")
# generate confusion matrix for test data
confusionMatrix(tree.pred.test, test$binned_score)

## Confusion Matrix and Statistics
## 
##           Reference
## Prediction (0,4] (4,6] (6,8] (8,10]
##     (0,4]      0     0     0      0
##     (4,6]      8   107    76      0
##     (6,8]      5   105   423     10
##     (8,10]     0     0     1      8
## 
## Overall Statistics
##                                          
##                Accuracy : 0.7241         
##                  95% CI : (0.6904, 0.756)
##     No Information Rate : 0.6729         
##     P-Value [Acc > NIR] : 0.001485       
##                                          
##                   Kappa : 0.3651         
##  Mcnemar's Test P-Value : NA             
## 
## Statistics by Class:
## 
##                      Class: (0,4] Class: (4,6] Class: (6,8] Class: (8,10]
## Sensitivity                0.0000       0.5047       0.8460       0.44444
## Specificity                1.0000       0.8418       0.5062       0.99862
## Pos Pred Value                NaN       0.5602       0.7790       0.88889
## Neg Pred Value             0.9825       0.8098       0.6150       0.98638
## Prevalence                 0.0175       0.2853       0.6729       0.02423
## Detection Rate             0.0000       0.1440       0.5693       0.01077
## Detection Prevalence       0.0000       0.2571       0.7308       0.01211
## Balanced Accuracy          0.5000       0.6733       0.6761       0.72153

Accuracy is 0.7241 for test set.

6.2.1 Data Pre-processing

First, we need to prepare our data for applying knn purpose. Dummy variables are required for categorical variables. We use a copy of our data, so we can still use our original data in the future.

library(FNN)
# Use model.matrix() to create dummy variables for country and content.
IMDB2 <- IMDB
IMDB2$country <- as.factor(IMDB2$country)
IMDB2$content <- as.factor(IMDB2$content)
IMDB2[,c("country_UK", "country_USA", "country_Others")] <- model.matrix( ~ country - 1, data = IMDB2)
IMDB2[,c("content_G", "content_NC17", "content_PG", "content_PG13", "content_R")] <- model.matrix( ~ content - 1, data = IMDB2)
# Select useful variables for future prediction.
IMDB2 <- IMDB2[, c(1,2,3,4,5,6,7,8,9,10,11,16,17,18,19,20,21,22,23,15)]
# Partition the data into training and validation sets.
set.seed(52)
train2 <- IMDB2[train.index, ]
valid2 <- IMDB2[valid.index, ]
test2 <- IMDB2[test.index, ]

Then we need to normalize our data.

# initialize normalized training, validation, test data, complete data frames to originals
train2.norm <- train2
valid2.norm <- valid2
test2.norm <- test2
IMDB2.norm <- IMDB2
# use preProcess() from the caret package to normalize predictors.
norm.values <- preProcess(train2[, -20], method=c("center", "scale"))
train2.norm[, -20] <- predict(norm.values, train2[, -20])
valid2.norm[, -20] <- predict(norm.values, valid2[, -20])
test2.norm[, -20] <- predict(norm.values, test2[, -20])
IMDB2.norm[, -20] <- predict(norm.values, IMDB2[, -20])

6.2.2 Find the best k

We will set k as 1 to 20, and build 20 different models. We calculate each model’s classification accuracy, and find the best k according to the highest accuracy.

# initialize a data frame with two columns: k, and accuracy.
accuracy.df <- data.frame(k = seq(1, 20, 1), accuracy = rep(0, 20))
# compute knn for different k on validation data.
for(i in 1:20) {
  knn.pred <- knn(train2.norm[, -20], valid2.norm[, -20],
                  cl = train2.norm[, 20], k = i)
  accuracy.df[i, 2] <- confusionMatrix(knn.pred, valid2.norm[, 20])$overall[1]
}
accuracy.df

##     k  accuracy
## 1   1 0.6671159
## 2   2 0.5916442
## 3   3 0.6940701
## 4   4 0.6792453
## 5   5 0.6954178
## 6   6 0.6913747
## 7   7 0.6886792
## 8   8 0.6873315
## 9   9 0.7142857
## 10 10 0.7061995
## 11 11 0.7021563
## 12 12 0.6873315
## 13 13 0.6940701
## 14 14 0.6846361
## 15 15 0.7008086
## 16 16 0.6886792
## 17 17 0.6913747
## 18 18 0.6886792
## 19 19 0.6873315
## 20 20 0.6927224

When k = 9, we get the highest accuracy: 0.7142857

6.2.3 Apply model on test set

# apply model on test set
knn.pred.test <- knn(train2.norm[, -20], test2.norm[, -20],
                cl = train2.norm[, 20], k = 9)
# generate confusion matrix for test data
accuracy <- confusionMatrix(knn.pred.test, test2.norm[, 20])$overall[1]
accuracy

##  Accuracy 
## 0.7456258

Test set accuracy: 0.7456258

6.3.1 Build Model

library(randomForest)
set.seed(53)
rf <- randomForest(binned_score ~ . -imdb_score, data = train, mtry = 5)
# Show model error
plot(rf)
legend('topright', colnames(rf$err.rate), col=1:5, fill=1:5)

The black line shows the overall error rate which falls below 30%. The red, green, blue and aqua lines show the error rate for bad, ok, good and excellent movies respectively. We can see that right now we’re much more successful predicting good movies. We cannot predict bad movies very well.

Let’s look at relative variable importance by plotting the mean decrease in Gini calculated across all trees.

# Get importance
importance <- importance(rf)
varImportance <- data.frame(Variables = row.names(importance), 
                            Importance = round(importance[ ,'MeanDecreaseGini'],2))

# Create a rank variable based on importance
rankImportance <- varImportance %>%
  mutate(Rank = paste0('#',dense_rank(desc(Importance))))

# Use ggplot2 to visualize the relative importance of variables
ggplot(rankImportance, aes(x = reorder(Variables, Importance), 
                           y = Importance, fill = Importance)) +
  geom_bar(stat='identity') + 
  geom_text(aes(x = Variables, y = 0.5, label = Rank),
            hjust=0, vjust=0.55, size = 4, colour = 'red') +
  labs(x = 'Variables') +
  coord_flip() + 
  theme_few()

From the plot, we see User_vote is a very important variable, while face_number, content and country are not so important.

6.3.2 Apply Model

set.seed(632)
# apply model on validation set
rf.pred.valid <- predict(rf, valid)
# generate confusion matrix for validation data
confusionMatrix(rf.pred.valid, valid$binned_score)

## Confusion Matrix and Statistics
## 
##           Reference
## Prediction (0,4] (4,6] (6,8] (8,10]
##     (0,4]      0     0     0      0
##     (4,6]      7   106    43      0
##     (6,8]      8   103   446     12
##     (8,10]     0     0     2     15
## 
## Overall Statistics
##                                           
##                Accuracy : 0.7642          
##                  95% CI : (0.7319, 0.7943)
##     No Information Rate : 0.6617          
##     P-Value [Acc > NIR] : 8.023e-10       
##                                           
##                   Kappa : 0.4547          
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: (0,4] Class: (4,6] Class: (6,8] Class: (8,10]
## Sensitivity               0.00000       0.5072       0.9084       0.55556
## Specificity               1.00000       0.9062       0.5100       0.99720
## Pos Pred Value                NaN       0.6795       0.7838       0.88235
## Neg Pred Value            0.97978       0.8242       0.7399       0.98345
## Prevalence                0.02022       0.2817       0.6617       0.03639
## Detection Rate            0.00000       0.1429       0.6011       0.02022
## Detection Prevalence      0.00000       0.2102       0.7668       0.02291
## Balanced Accuracy         0.50000       0.7067       0.7092       0.77638

Accuracy is 0.7642 for validation set.

set.seed(633)
# apply model on test set
rf.pred.test <- predict(rf, test)
# generate confusion matrix for test data
confusionMatrix(rf.pred.test, test$binned_score)

## Confusion Matrix and Statistics
## 
##           Reference
## Prediction (0,4] (4,6] (6,8] (8,10]
##     (0,4]      0     1     0      0
##     (4,6]      8   114    51      0
##     (6,8]      5    97   447     10
##     (8,10]     0     0     2      8
## 
## Overall Statistics
##                                           
##                Accuracy : 0.7658          
##                  95% CI : (0.7337, 0.7958)
##     No Information Rate : 0.6729          
##     P-Value [Acc > NIR] : 1.796e-08       
##                                           
##                   Kappa : 0.4515          
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: (0,4] Class: (4,6] Class: (6,8] Class: (8,10]
## Sensitivity              0.000000       0.5377       0.8940       0.44444
## Specificity              0.998630       0.8889       0.5391       0.99724
## Pos Pred Value           0.000000       0.6590       0.7996       0.80000
## Neg Pred Value           0.982480       0.8281       0.7120       0.98636
## Prevalence               0.017497       0.2853       0.6729       0.02423
## Detection Rate           0.000000       0.1534       0.6016       0.01077
## Detection Prevalence     0.001346       0.2328       0.7524       0.01346
## Balanced Accuracy        0.499315       0.7133       0.7165       0.72084

Accuracy is 0.7658 for test set.