03 / Independent ML project · 2025
Credit decision lab with calibrated evidence
Chronological validation, calibrated probabilities, decision cost and subgroup diagnostics.
The project was rebuilt after audit to remove circular target proxies and exaggerated production claims. It demonstrates how probabilities become decisions while stating that it emulates synthetic approvals rather than predicting real defaults.
Evidence register
0.9669
ROC-AUC
Final chronological holdout of 4,000 synthetic rows
0.0606
Brier score
Calibrated logistic reference run
0.1445
cost / case
Validation-selected 5:1 decision cost
01 / Problem
A ranking metric does not decide whom to approve. A decision workflow also needs calibrated probabilities, explicit error costs, temporal validation and diagnostics for uneven outcomes.
02 / Approach
A leakage-controlled pipeline excludes direct decision proxies, fits on earlier synthetic applications, calibrates on a disjoint period and selects a threshold from a 5:1 cost matrix before one final holdout evaluation.
03 / Outcome
The repository produces a versioned model artifact, evaluation and subgroup reports, a strict inference API and a reproducible reference run over 20,000 synthetic applications.
System trace
How the evidence is produced.
Reframed and rebuilt the workflow around leakage control, calibration, decision cost, subgroup diagnostics, tested inference and honest evidence boundaries.
- 01Synthetic dated applications → proxy exclusions
- 02Chronological fit → candidate models
- 03Disjoint calibration → cost-sensitive threshold
- 04Final holdout → reports, artifact and API
Validation scope
Six automated tests pass. The final 4,000-row holdout reports ROC-AUC 0.9669, AP 0.9097 and Brier 0.0606; the 0.835 threshold yields 33 false approvals and 413 missed approvals under the documented cost model.
Known limitation
The data and approval mechanism are synthetic, so the system does not estimate real default risk. A 14.3-point age-band selection-rate spread is disclosed as a risk requiring contextual investigation, not proof of fairness.
What is inspectable
- Direct risk scores, pricing outputs and audit attributes are excluded from training.
- Calibration and threshold selection never inspect the final holdout.
- Schema tests, CI, a data card and explicit subgroup tables make claims inspectable.
Next proof to add
- 01Replace the synthetic target with governed default outcomes and temporal cohorts.
- 02Add confidence intervals, drift monitoring and documented adverse-action review.
- 03Validate costs and fairness criteria with risk, legal and affected stakeholders.
Main stack