Tributary
What it is
A simulated two-sided marketplace sells personal-loan leads through a six-tier waterfall auction. Its operational data is generated from public datasets (iPinYou bid logs, LendingClub applications, Criteo uplift) and deliberately fractured into three cloud systems with no shared key, the way real marketplace data fractures. The case study asks the questions the business would ask — which channels earn their spend, what a duplicate consumer costs, where auction floors should sit, who a nurture campaign should reach — and answers them: identity modeling scored against a hidden ground-truth crosswalk, calibrated propensity and price models, a counterfactual reserve-price replay, and an uplift experiment read at its actual power.
How it was built
The build was directed by me and executed with AI agents inside a harness: a charter and standing instruction set, a machine-readable index of the repository that agents load at session start, and a mandatory record for every event that matters — a decision between real alternatives, a correction of an agent’s misunderstanding, a prompt that shaped an artifact. Every commit carries a provenance trailer classifying it as human-authored, human-directed, or agent-executed, so git log is the contribution ledger. A phase is not done when the code works; it is done when the logs are current, the index validates, and provenance is recorded.
What it demonstrates
Each row names a case-study artifact, the business decision it informs, the skill it shows, and the resume claim it evidences. Every figure traces to a committed artifact in the repository — model metrics, the validation run, the entity-resolution scorecard, or the dashboard aggregates.
| Artifact | Decision it informs | Skill shown | Resume claim it evidences | Where |
|---|---|---|---|---|
| Problem-framing table: decision, metric, counterfactual, data, method for each of eight business questions | All eight, before any method is chosen | Translating a business problem into a data-science problem | Leadership & communication (recommendations actionable for executives and engineers) | Overview |
| Identity modeling: probabilistic linkage where identity is fuzzy, deterministic where an exact key exists, clustered at the 0.9998-precision operating point; link F1 0.879, dedupe F1 0.873; 95.3% of auction events joinable, from 0%; difficulty deliberately retuned into the 0.8–0.9 band | Every per-consumer figure the business reports | Record linkage as modeling: operating points, precision–recall trade-offs, scoring against ground truth, error propagation into a decision | Fraud detection at scale (identity-graph engine); Closed-loop funding feedback pipeline (privacy-preserving matching on hashed identifiers) | Identity modeling |
| Duplicate-consumer cost: $12.0M paid by the same buyer for consumers bought within the prior 30 days (5.7% of revenue), $48.3M across any buyer; window curve from 1 to 90 days; 20% of consumers produce 53% of revenue | Suppressing or re-pricing recent repeats | Turning an identity result into dollars; stating the policy question with its missing experiment | Outbound messaging optimization engine (recency-based value thresholds and suppression) | Findings |
| Channel attribution: ROAS 0.94x to 3.06x, blended 1.94x on $53.8M; cost per sold lead $57 to $183; the click ranking inverts once revenue joins through the auction | Where the next acquisition dollar goes; whether display continues | Attribution design (cohort vs last-touch), recognizing right-censoring, refusing a budget call the data cannot support and specifying the test that would | Marketing analytics (conversion and customer-path reporting; campaign targeting) | Findings |
| Nurture holdout read-out: +0.109pp application lift with a 95% interval spanning zero (-0.105pp to +0.324pp); the pooled effect would need about 6.1M contacts, so the read is segment-level and the follow-up is sized at 134,179 contacts at 50/50 | Whether to claim a program lift; how to size the follow-up test | Experiment design and honest reads; power arithmetic; decision rules under underpowered evidence | Experimentation (power analysis, rigorous A/B design) | Findings |
| Sale propensity: gradient-boosted model under an explicit leakage contract and an 8/2/2-month temporal split, calibrated to an expected calibration error of 0.0017 (from 0.0031 uncalibrated); the tier head scores 1.096 log-loss against a 1.220 baseline | Pre-auction lead decisioning and routing priority | Calibration discipline; leakage contracts; temporal validation; stating what does not transfer from synthetic data | Machine learning (lead sale-propensity models in production) | Models and experiments |
| Winning-price landscape: discrete-time survival over the tier ladder; expected-value rank correlation 0.859 against 0.269 for the group table, and the top decile captures 33.3% of revenue against 14.7%. A Tobit specification test, kept as evidence rather than discarded, prices unsold leads at a median $0.18 against $32.25 of logged sub-reserve demand | Lead valuation for pricing and segment-level reserves | Censored-data modeling; choosing the framing from the censoring mechanism | Experimentation (owns real-time bidding and auction mechanics); Reinforcement learning for lead routing (auction simulator from buyer behavior) | Models and experiments |
| Reserve-floor optimization: exact counterfactual replay of 2.4M logged waterfalls recommends reshaping the schedule rather than moving its level, worth +2.02% revenue per lead (95% CI +1.96% to +2.08%), about $4.2M a year at the logged volume; a five-seed engine re-run realizes +2.98% (range +2.88% to +3.15%), and the gain reaches break-even if buyers give back about 19% of the tier-1 increase | Where reserve floors sit; how to stage and test the change | Auction mechanics; counterfactual policy evaluation; validating an offline estimate against a simulator; quantifying assumption risk, not just sampling error | Reinforcement learning for lead routing (off-policy evaluation, policy gated behind a live test); Experimentation (auction mechanics) | Models and experiments, Strategy memo |
| Uplift model: cross-fitted T-learner on the action space, capturing 68% of the oracle ceiling, with response targeting shown to score below random; a tie-breaking artifact that had inflated the oracle was caught and removed. About 1.77M messages a year avoided | Whom to message and whom to stop messaging | Uplift modeling; power-driven estimator choice; covariate selection argued from the action space | Outbound messaging optimization engine (nightly scoring and suppression) | Models and experiments |
| Strategy memo: five recommendations with expected value, uncertainty, a risk register, a staged rollout with a stopping rule, and the order to test them in — floors first, because their expected value is two orders of magnitude larger than the send policy’s | Which change ships first, and how it is tested | Decision memos for executives; prioritizing tests by expected value; saying what is not yet decision-grade and why | Leadership & communication; Revenue impact (taking a vertical through prediction to automated optimization) | Strategy memo |
| Calibration judgment: a statistically valid elasticity estimate rejected as non-transferable; a verification gate that falsified its own parameter kept on record | Whether simulated statistics can be trusted at all | Knowing when a good estimate does not transfer; verification gates over trust | No single bullet claims this; it supports the M.S. Systems Science line (applied modeling of systems) | How this was built |
| The harness: charter, standing instructions, machine-readable repository index loaded at session start, mandatory records, per-commit provenance | How agents are made reliable enough to build the above | Designing guardrails and a version-controlled knowledge base that humans and AI tooling both read | AI agent guardrails & knowledge base | How this was built |
| Three-silo cloud build on free tiers, with a one-cent bill, a 10.9x scan reduction from partitioning, and free-tier fit by measurement; six static dashboards | None directly: how the lab was built | Data engineering and cost discipline, presented as a supporting skill | Analytics infrastructure (analytics and BI platform, real-time dashboards) | Appendix, Findings |
Resume claims the case study does not evidence, left unmapped on purpose: department creation, the measured revenue impact of the three verticals, fraud detection (rules plus unsupervised ML), KPI alerting from time-series forecasts, competitor intelligence, merchant risk modeling and network visualization, and the research-intern data mining. Those stand on the resume alone.
Everything in the table reproduces from the repository with one seeded command; see Run it yourself.