ML Decision Tree

ML Decision Trees: Classification & Splitting Metrics

Decision Trees are non-parametric supervised learning models that partition feature space into recursive axis-aligned decision boundaries using metric splits like Gini Impurity or Entropy.

Decision Tree Splitting Flow

flowchart TD
    Root["Root Node: Feature X <= 2.5?"] -- "Yes" --> Left["Leaf Node: Class 0 (Pure)"]
    Root -- "No" --> Right["Sub-Node: Feature Y <= 50.0?"]
    Right -- "Yes" --> Leaf1["Leaf Node: Class 1"]
    Right -- "No" --> Leaf2["Leaf Node: Class 0"]

Impurity Metrics

• Gini Impurity: $G = 1 - \sum_{i=1}^{C} p_i^2$ (Measures likelihood of misclassifying a randomly chosen element).
• Entropy: $H = -\sum_{i=1}^{C} p_i \log_2(p_i)$ (Information theory metric measuring disorder).

Practical Code Example

import numpy as np

def calculate_gini_impurity(y: np.ndarray) -> float:
    """Calculate Gini Impurity for a target class array."""
    if len(y) == 0:
        return 0.0

    _, counts = np.unique(y, return_counts=True)
    probabilities = counts / len(y)
    gini = 1.0 - np.sum(probabilities ** 2)
    return float(gini)

if __name__ == "__main__":
    # Pure node (all class 1)
    pure_labels = np.array([1, 1, 1, 1])
    # Impure split node (half class 0, half class 1)
    impure_labels = np.array([0, 0, 1, 1])

    print(f"Gini Impurity (Pure Node)   : {calculate_gini_impurity(pure_labels):.4f}")
    print(f"Gini Impurity (Impure Node) : {calculate_gini_impurity(impure_labels):.4f}")

Best Practices & Gotchas

• Control Overfitting with max_depth: Unconstrained decision trees grow until every leaf node is pure, leading to extreme overfitting. Constrain tree depth with max_depth or min_samples_split.
• Feature Importance: Decision trees provide built-in feature importance rankings based on total impurity reduction per feature.
• Insensitive to Feature Scaling: Decision trees operate on single-feature threshold splits, making feature scaling unnecessary.

Self-Check Challenge

Calculate the Gini Impurity of a label array [0, 1, 1, 1] using the calculate_gini_impurity() function.