Brown Researchers Built TempusBench to Fix AI Forecasting's Broken Leaderboards
A team of academic researchers released TempusBench, an open framework that stress-tests time-series foundation models on tasks their pretraining data has never seen.
ProTakeaways
- Researchers released TempusBench, an open evaluation framework for time-series foundation models.
- Uses new datasets deliberately excluded from existing TSFM pretraining corpora to avoid data leakage.
- Ships 25 univariate and 23 multivariate tasks probing non-stationarity and seasonality.
- Applies a standardized hyperparameter tuning protocol uniformly across foundation and classical models.
- Bundles Chronos, TimesFM, Moirai, Lag-Llama, Toto, XGBoost, ARIMA, LSTM, and more.
- Code and TensorBoard-based visualization available at Smlcrm/TempusBench.
Time-series forecasting is having its foundation-model moment. Models like Chronos, TimesFM, Moirai, and Lag-Llama promise to do for temperatures, sales, and sensor readings what GPT did for text. But how do you know which one actually works on your data? A group of academic researchers, led by authors including Denizalp Goktas and Amy Greenwald at Brown, argue that the field has been grading these models on a broken exam, and they are releasing TempusBench, an open-source evaluation framework built to fix it.
Broken scoreboards, leaky exams
Existing time-series foundation model (TSFM) benchmarks suffer from four issues that make comparisons close to meaningless. Datasets like M3 are old, poorly documented, and worst of all, likely sit inside the pretraining corpora of the very models being tested. That is the forecasting equivalent of open-book cheating.
The authors point to three more problems. Benchmark tasks tend to slice models along horizon length or domain, but ignore statistical properties like non-stationarity and seasonality that determine whether a model will work on your series. Domain-specific baselines such as XGBoost rarely get the same hyperparameter tuning budget as the foundation models they are compared against. And once results are in, there is almost nothing to help you visualize where a model wins or loses.
What TempusBench actually ships
The framework, hosted at Smlcrm/TempusBench