Knowledge Graph — Coursera Notes › Academic disciplines › Information Technology / Computer Science › Machine Learning

Ensemble methods

concept · part of Machine Learning

Ensemble methods combine multiple machine learning models to improve predictive performance, accuracy, and robustness while reducing overfitting. They work by aggregating predictions from diverse models, often outperforming single models. Key techniques include bagging, boosting, and stacking. Trade-offs include higher computational cost and reduced interpretability, which can be addressed with tools like SHAP and LIME.

Bagging (bootstrap aggregating) reduces variance and improves stability by training multiple models on different bootstrapped subsets of data (random sampling with replacement). Predictions are aggregated: majority voting for classification, average for regression. Random forest is a popular implementation, building decision trees on bootstrapped datasets with random feature subsets. Example: credit scoring uses bagging to aggregate outputs from multiple decision trees, minimizing impact of noisy data.

Boosting sequentially trains models, each correcting errors of its predecessor, with final output as a weighted combination. AdaBoost assigns higher weights to misclassified samples; Gradient Boosting Machines (GBMs) use gradient descent to optimize a loss function. Effective for improving weak learners (e.g., shallow decision trees) and reducing bias. Example: fraud detection uses boosting to iteratively refine predictions, minimizing false negatives.

Stacking combines multiple base models via a meta-learner (e.g., linear regression or neural network). Base model predictions become input features for the meta-learner, which produces the final output. Unlike bagging and boosting, stacking leverages diverse models to capture complex patterns. Example: e-commerce recommendation systems integrate collaborative filtering, content-based filtering, and neural network predictions for improved personalization.

Ensemble methods can also be combined with interpretability tools like SHAP and LIME to explain predictions. SHAP (SHapley Additive exPlanations) is a game-theoretic approach that assigns each feature an importance value for model interpretability. Example: in healthcare predicting heart disease risk, SHAP highlights influential factors like cholesterol level or age.

LIME (Local Interpretable Model-agnostic Explanations) creates a locally interpretable model around a specific prediction. Example: in text classification, LIME indicates which words heavily influenced a review's classification as positive or negative.

Ensemble methods generally reduce overfitting compared to single models, but it's important to ensure diversity among base models to avoid correlated errors.

Training multiple models and aggregating predictions requires significant computational resources. Parallelizing training processes can mitigate this issue.

Inside Ensemble methods (1)

• Boosting — Boosting is an ensemble machine learning technique that sequentially trains weak learners, each focusing on correcting the errors of its predecessor, to create a strong predictive model.

This is the text view of an interactive 3D knowledge graph — open this page with JavaScript enabled to explore it visually.