On this page (5 sections)
Content reviewed: 2026-10-13. Examples are hypothetical.
A prediction model for event contracts should produce a probability for a precisely defined outcome using only information available at the forecast time. Its value comes from performance on later, unseen cases, not how closely it explains the examples used to build it.
Define the observation and label
Choose one row per forecast opportunity and write the outcome rule before collecting data. Record the issue time, horizon, features, predicted probability and eventual label. If the contract uses an initial economic release, label it with that vintage rather than a revised value. BLS documents that published series can change through several revision processes.[2]
For related contracts, keep an event identifier. Ten snapshots of one election are not ten independent elections. They should not be split randomly between training and evaluation in a way that lets the model learn the same event's result indirectly.
Establish a simple baseline
A historical base rate or a contemporaneous market probability gives you a comparison before adding complexity. Start with a small set of defensible features. Preserve the timestamp of each input so a later database revision cannot quietly introduce information that was unavailable when the prediction would have been made.
Use a chronological evaluation: train on earlier observations, choose model settings on a later validation period, and leave a final period untouched until the process is fixed. Repeat with rolling windows if the event population changes over time.
Score probabilities rather than just winners
Proper scoring rules reward honest probabilistic forecasts under their assumptions.[1] For binary outcomes, a common Brier-loss convention is:
mean((forecast probability - outcome)^2)
With probabilities 0.8 and 0.3 and outcomes 1 and 0, the mean is (0.04 + 0.09) / 2 = 0.065. State this one-outcome convention because some implementations sum losses over both binary categories and produce twice the value.
For log loss, use -mean(y × log(p) + (1-y) × log(1-p)). Assigning probability zero to an event that occurs gives infinite loss; software clipping changes that rule, so document any clipping threshold rather than hiding it.[1]
Check calibration and trading performance separately
Group comparable forecasts into probability bands and compare their average forecast with their observed outcome frequency. A small band sample can be noisy. Also examine performance by event type and horizon so a strong aggregate does not conceal a weak subgroup.
Then simulate trading with historical executable quotes, fees and quantity constraints. A model can improve forecast scores without producing a profitable strategy after costs. Keep the position-sizing rule and execution assumptions fixed during the evaluation.
Save model versions, input data vintages and every forecast, including decisions not to trade. That record makes later improvements measurable and prevents a series of retrospective explanations from being mistaken for a tested model.
References
- Tilmann Gneiting and Adrian E. Raftery. Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association 102(477), 359-378, March 2007. DOI: 10.1198/016214506000001437.
- U.S. Bureau of Labor Statistics. Revisions. Accessed October 13, 2026.