Knowledge Graph — Coursera Notes › Academic disciplines › Information Technology / Computer Science › Machine Learning

Model evaluation

concept · part of Machine Learning

TensorFlow's evaluate method simplifies evaluation. PyTorch requires manual computation of accuracy. Both frameworks achieve around 70% test accuracy on CIFAR-10 after 10 epochs.

Test accuracy is calculated to measure how well the model predicts diabetes for unseen patients, achieving approximately 75% accuracy.

After fine-tuning, the Trainer's predict method generates predictions on the test set. Accuracy and weighted F1 score are computed using sklearn.metrics.accuracy_score and f1_score with average='weighted'.

predictions = trainer.predict(test_dataset)
preds = predictions.predictions.argmax(-1)
labels = test_dataset['label']
accuracy = accuracy_score(labels, preds)
f1 = f1_score(labels, preds, average='weighted')

The test set must be distinct from the validation set: the validation set is used during fine-tuning for hyperparameter tuning and monitoring, while the test set is used only after fine-tuning for final evaluation to obtain an unbiased measure of generalization.

For balanced datasets use accuracy. For imbalanced use precision, recall, F1, or ROC-AUC. When false positives costly use precision and specificity. When false negatives costly use recall. When both matter use F1.

Inside Model evaluation (4)

• Evaluation metrics — Key metrics for evaluating fine-tuned models include accuracy, precision, recall, F1 score, and confusion matrix.
• Loss — Loss measures alignment between predictions and actual labels.
• ROC-AUC — ROC-AUC measures trade-off between true positive rate (recall) and false positive rate (1-specificity) across thresholds.
• Specificity — Specificity (true negative rate) measures proportion of actual negatives correctly identified: TN/(TN+FP).

This is the text view of an interactive 3D knowledge graph — open this page with JavaScript enabled to explore it visually.