The Curse of Dimensionality

</aside>

<aside> 💡

Overfitting

</aside>

<aside> 💡

Training, Testing, and Validation Sets

Training Set

Used to train the model by adjusting its parameters based on input features and corresponding outputs.
Typically the largest portion of the dataset (60-80% of total data).
The model learns patterns from the training set through optimization algorithms (e.g., gradient descent in deep learning).
If you’re building a spam classifier, the training set would contain labeled emails where the model learns to distinguish spam from non-spam.

Validation Set

Used to tune hyperparameters and evaluate the model during training, helping to prevent overfitting.
Typically 10-20% of the total data.
The model is trained on the training set, and then its performance is checked on the validation set to fine-tune aspects like learning rate, number of layers, or regularization parameters.
In a neural network, you might adjust the dropout rate based on how well the model performs on the validation set.

Testing Set

Used to assess the final performance of the trained model on unseen data.
Typically 10-20% of the total dataset.
After finalizing the model (based on validation performance), it is tested on the test set to check how well it generalizes to completely new data.
If a spam detection model achieves high accuracy on the test set, it means the model is ready for real-world deployment.

Need to consider

Avoid Data Leakage – Ensure that test data is not used in training or hyperparameter tuning.
Cross-Validation – Techniques like k-fold cross-validation can be used when the dataset is small to ensure better model evaluation.
Stratified Splitting – If dealing with imbalanced classes, stratified sampling ensures that all sets maintain the same class distribution.

** Data leakage occurs when information from outside the training dataset is used to create the model, leading to overly optimistic performance during training but poor generalization on unseen data. This results in models that appear highly accurate in development but fail in real-world scenarios.

*** Types of leakage?

→ Train-Test Contamination: Happens when test data is accidentally included in the training set. If future stock prices are included in the training data while predicting stock trends, the model will learn patterns it wouldn’t have access to in real-world conditions.

→ Feature Leakage (Target Leakage): Occurs when features used for training include information that wouldn’t be available at prediction time. In a loan approval model, using a feature like "whether the loan was paid off" to predict loan default would make the model perform unrealistically well.

*** How Prevent? ensure proper data splitting; avoid using future information; use cross-validation correctly

*** Why serious? (1) cause poor generalization on real-world data; (2) can result in costly mistake in field like finance, healthcare and cybersecurity

Confusion Matrix

The Receiver Operating Characteristic (ROC) curve is a graphical representation used to evaluate the performance of a classification model, particularly in binary classification problems. It shows the trade-off between True Positive Rate (TPR, Sensitivity) and False Positive Rate (FPR) at various classification thresholds.

The x-axis represents FPR (1 - Specificity), and the y-axis represents TPR (Sensitivity)

A perfect classifier would have a point at (0,1), meaning zero false positives and 100% true positives. The closer the curve is to the top-left corner, the better the model performs.

The closer the curve is to the top-left corner, the better the model, as it achieves a high TPR while keeping the FPR low.

The Area Under the ROC Curve (AUC-ROC) quantifies the overall performance of a classification model. It is calculated as the area under the Receiver Operating Characteristic (ROC) curve, which plots the True Positive Rate (TPR) against the False Positive Rate (FPR) at different thresholds.

</aside>

Bagging and Boosting