Scoring an already frozen forecast against revised GDP can be legitimate. Leakage arises when later information enters the inputs, training or selection that supposedly preceded that forecast. The distinction matters because a changed score may reveal a different evaluation target rather than a dishonest information experiment.
- Later GDP can be a legitimate evaluation target when chosen explicitly.
- Revised predictors or training outcomes can leak information into an earlier forecast.
- Changing the target vintage can alter model rankings because initial errors and revisions can interact.
Later scoring is a different act from earlier forecasting
The information audit places predictors, training outcomes and selection decisions on the forecast clock. If later revisions enter any of those at an earlier origin, the prediction may use unavailable information. The target used afterward to score the recorded forecast belongs on a separate clock. Choosing a later GDP vintage as that target is legitimate when the question is stated in advance. The New York Fed evidence motivates the information audit, while this separation explains why later scoring alone need not constitute leakage. [3]
The New York Fed's study of macroeconomic data and Treasury returns documents how revised observations can exaggerate the information available historically. Revisions account for a substantial share of apparent predictive power in its analysis. The problem concerns what investors could know before a decision. The paper is an investment information study, so it does not rank GDP nowcasting algorithms or establish that every later GDP evaluation target introduces the same information mismatch. [3]
A frozen prediction can face different GDP targets
The Bank of England's Uncertain Kingdom working paper models early GDP releases and subsequent revisions using a release augmented dynamic factor approach. It improves first release forecasts relative to a standard model and finds that economic and survey data help predict the first revision round. Later incremental predictive content is largely exhausted in its setting. The paper therefore supplies evidence that the release process can contain forecastable structure. It does not imply that every revision, in every economy or vintage, is predictable. [4]
Define error as the realised GDP estimate minus the frozen forecast. Let the revision be later GDP minus its first release. The later error then equals the first release error plus that revision. Squaring adds the squared revision and twice its product with the original error. The product differs across forecasts, so a target change can alter rankings in either direction. This is an accounting identity, motivated by distinct release targets, rather than an estimated revision effect. [4]
A release calendar is not a complete vintage archive
The ECB describes a forecasting evaluation targeted at the first GDP flash release. Its framework also distinguishes news from historical data revisions and parameter estimation changes. For inputs without historical vintages, it uses final data while simulating the earlier publication lag. That supplies a practical partial reconstruction of the information calendar, but it does not recover every original numerical observation. The limitation belongs in the interpretation of the results, even when the calendar and evaluation target are carefully specified. [7]
The vintage audit asks separately whether a value was available and whether its release date was available. A simulated publication lag can prevent an indicator from entering before its scheduled release while still allowing later revised values to stand in for the original ones. Those controls address different problems. The ECB's disclosed convention makes the distinction concrete. A report should identify which series have actual historical snapshots and which rely on reconstruction, rather than calling every lagged dataset a complete record of historical information. [7]
A foundational Federal Reserve paper formalises forecast updates as releases arrive. It separates the contribution of publication timing from economic content, computing news relative to the evolving information set. That makes the release calendar part of the nowcasting problem. Its historical findings about particular indicators are not a ranking of their importance today. The underlying lesson is that information must be evaluated relative to what was already known, rather than treating the latest accessible database as the earlier forecaster's dataset. [2]
Freeze training as well as the incoming indicators
The Bank of England's staggered combination MIDAS approach uses the relationship between monthly and quarterly GDP and changes the emphasis on survey and hard data through the release cycle. It reports improvements over pooled MIDAS and several benchmarks. This reinforces the need to distinguish origins within the same quarter: a later origin can incorporate observations unavailable earlier. The abstract supports the model design and comparative findings, while leaving the full benchmark and score definitions to the paper. It does not certify a separate historical dataset. [5]
A reviewable forecast receipt records the origin, release timestamps, input snapshots, transformations and model specification. It asks whether each observation was available before the cutoff and preserves the forecast before later releases arrive. The evolving information framework in the Fed paper motivates this sequence. Keeping the record would make a historical information claim reviewable. It would not ensure that the transformations were sensible or that the resulting estimate forecast GDP well; those questions require their own evaluation. [2]
The training record also freezes the GDP outcomes available for estimation, tuning choices and the refitting rule at each origin. The Bank of England's calendar dependent modelling makes those choices relevant because useful information changes as the quarter progresses. If the training process silently replaces early outcomes with later revisions, a correctly dated predictor archive alone cannot resolve the information mismatch. The audit should therefore follow the estimation inputs as well as the newly arriving indicators, while keeping legitimate later scoring separate. [5]
What the two receipts allow a reviewer to conclude
The IMF's nowcasting comparison identifies strong traditional benchmarks and warns that complex methods can overfit short GDP histories. That warning remains relevant after a vintage audit passes. Correctly dated inputs establish an information boundary; they do not establish that the fitted relationship generalises. Conversely, an impressive benchmark result does not certify the input archive. Information validity and forecast performance are separate tests, each capable of failing while the other appears satisfactory. [6]
The Federal Reserve's Bells and Whistles paper uses information sets designed to approximate historical conditions and compares feature contributions across several scoring objectives. Some features help one criterion without consistently helping another. That evidence separates model design from the question being scored. It also supports retaining the study's qualification about reconstructed historical information. A reader should not infer complete vintage certification from a predictive comparison, nor assume that improving a point forecast means improving its full uncertainty distribution. [1]
If correcting unavailable inputs changes a score, first describe the change in the information experiment. A matched traditional benchmark would help test whether the remaining advantage comes from modelling rather than data access. The IMF comparison supplies a reason to take those alternatives seriously. The correction does not itself identify the winning algorithm. It establishes which historical knowledge claim is being tested, leaving the forecast result to a comparison under the same permitted inputs. [6]
For the final report, disclose the target vintage alongside the score and repeat the comparison across explicitly chosen objectives when relevant. The Fed feature study demonstrates why those criteria can disagree. Our conditional interpretation is that a model might be useful for the first release and less useful for a later economic estimate, or vice versa. Preserving both questions avoids turning a changed ranking into an unsupported claim of data leakage or universal superiority. The missing task is to run the chosen comparisons. [1]
Editorial assessment. The audit reaches two separate judgments: whether the prediction respected its information date, and whether the scoring target answers the declared question. A later target can change rankings without altering either forecast because its revision interacts differently with their original errors. Under an explicit later vintage objective, rescoring a frozen forecast deserves acceptance. Replacing unavailable training or predictor values at the earlier origin deserves a different judgment. The receipts make that distinction reviewable without promising that either forecasting task produces an investment edge.
Evidence scope. The studies use genuine historical vintages and partial reconstructions for different forecasting or investment tasks. The ECB's replayed publication lags do not recover unavailable original values. The error identity is an algebraic explanation, not an estimated covariance result. No current GDP estimate, vintage backtest or trading strategy is evaluated, and the audit does not certify a model's predictive performance.
Sources and further reading
- Bells and Whistles of Nowcasting Models: Which Matter and When?
- Nowcasting GDP and Inflation: The Real-Time Informational Content of Macroeconomic Data Releases
- Forecasting through the Rear-View Mirror: Data Revisions and Bond Return Predictability
- Uncertain Kingdom: nowcasting GDP and its revisions
- Nowcasting GDP at the Bank of England: a Staggered-Combination MIDAS approach
- GDP Nowcasting Performance of Traditional Econometric Models vs Machine-Learning Algorithms: Simulation and Case Studies
- Short-term forecasting of euro area economic activity in an uncertain world