Protocol
Model Selection Without Test Leakage
Write a comparative model-selection report where the most accurate model need not be the deployed choice. Defend the tradeoff under a stated latency or interpretability constraint.
35–60 active hours60–90 min sessionsIntermediate
Mastery contract
Core protocol · Standard 1.0 · Write a comparative model-selection report where the most accurate model need not be the deployed choice. Defend the tradeoff under a stated latency or interpretability constraint.
Loading your learning records…
Reasoning — not yet passed
Use saved learning curves to distinguish underfitting from overfitting and test an alternative explanation such as label noise.
Reliable implementation — not yet passed
For grouped observations, enforce entity separation and nested selection with leakage tests; contrast ordinary random K-fold with grouped validation using the same model and budget.
Reproducible experiment — not yet passed
Report all candidates, identical search budgets and uncertainty across outer folds without treating correlated folds as independent samples.
Defense and handoff — not yet passed
Defend a model and feature-set choice using held-out evidence, complexity, cost and the limitations of the selection procedure.
All four criteria must pass. Activity completion does not satisfy them.
An independent reviewer reruns your work and varies something you did not rehearse: a fresh input, a different working directory, or a declared edge case.
Pilot decisions are provisional, not external credentials. For an appeal or sensitive artifact, contact your designated pilot operator with the submission ID. Export your assessment history.
What you will do
6 total
Explain bias/variance using repeated synthetic samples; identify where test-set information can enter feature selection or hyperparameter tuning.
Implement nested cross-validation with preprocessing and feature selection inside each fold; persist every split and search configuration.
Compare 3 model capacities under the same folds and search budget; plot training/validation learning curves at 4 training-set sizes.
Debug globally selected features and repeatedly consulted holdout scores; rebuild the experiment with a new untouched evaluation set.
Write a 500–900 word technical report linking the reasoning, code, measurements, and limitations; include the project decision below.
Assess all 4 mastery criteria against saved artifacts, request an independent review, and repeat each failed criterion on a fresh example.
Essentials
A reading session alone never satisfies a mastery criterion.
Store source references, assumptions, and artifact paths beside every result.
Keep an untouched check case that differs from the worked example.
Record all attempts, including failures and results that contradict your prediction.
Use the stated comparison conditions; document every deviation before drawing a conclusion.
Protocol guardrails
Do
- +State the expected result before running the comparison.
- +Keep one minimal reproducible failing case when debugging.
- +Record environment versions and the exact command used.
- +Compare explanations with saved intermediate values.
- +Ask a reviewer to challenge the weakest assumption.
Don't
- ×Do not copy a worked solution and present it as an independent implementation.
- ×Do not tune against held-out evaluation outcomes.
- ×Do not report only the best seed or discard inconvenient runs.
- ×Do not equate elapsed hours or a completed run with a passed assessment.
- ×Do not conceal reduced-scale experiments behind claims about the original full-scale result.
Protocol authorship
Written by NuthinButta as instructional design. The teaching sources are the official references linked in each milestone; the exercises, workload and pass thresholds are ours, not their authors'.