ML Train/Test

ML Train/Test Split: Validation & Preventing Data Leakage

Evaluating a model on the same data used during training leads to overfitting. Splitting data into separate Training and Testing sets evaluates real-world generalization performance.

Train/Test Data Partitioning

flowchart LR
    Data["Full Dataset (100%)"] --> Train["Training Set (80%)
Used to fit model weights"]
    Data --> Test["Testing Set (20%)
Held-out for unbiased evaluation"]

Key Metrics

• MSE (Mean Squared Error): $rac{1}{N} \sum (y - \hat{y})^2$
• RMSE (Root Mean Squared Error): $\sqrt{ ext{MSE}}$
• R-squared ($R^2$): Variance explained on held-out test data.

Practical Code Example

import numpy as np
from typing import Tuple

def manual_train_test_split(
    X: np.ndarray, 
    y: np.ndarray, 
    test_ratio: float = 0.2, 
    seed: int = 42
) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
    """Randomly partition data into train and test splits."""
    np.random.seed(seed)
    n_samples = X.shape[0]
    shuffled_indices = np.random.permutation(n_samples)

    test_set_size = int(n_samples * test_ratio)
    test_indices = shuffled_indices[:test_set_size]
    train_indices = shuffled_indices[test_set_size:]

    return X[train_indices], X[test_indices], y[train_indices], y[test_indices]

if __name__ == "__main__":
    X_data = np.arange(100).reshape(50, 2)
    y_data = np.arange(50)

    X_train, X_test, y_train, y_test = manual_train_test_split(X_data, y_data, test_ratio=0.2)

    print(f"Full Dataset shape : {X_data.shape}")
    print(f"Training Set shape  : {X_train.shape} ({len(y_train)} samples)")
    print(f"Testing Set shape   : {X_test.shape} ({len(y_test)} samples)")

Best Practices & Gotchas

• Prevent Data Leakage: Never allow test set information to leak into training preprocessing (e.g. scaling or missing value imputation).
• Stratified Splits for Classification: When classification target classes are imbalanced, use stratified splitting to maintain class proportions.
• Cross-Validation: For small datasets, use $K$-Fold Cross Validation instead of a single static train/test split.

Self-Check Challenge

Split a dataset of 100 samples into 80% train and 20% test using a random seed, and check the length of both splits.