Skip to main content
Back

Harvard and Cambridge's Weighted ERM Finds Hidden Breakpoints in High-Dimensional Data

A new sample-weighting technique for detecting where regression models break down finds change points more accurately than existing state of the art methods.

Pro

  • New method Weighted ERM finds change points in high-dimensional GLMs via sample-weighted empirical risk minimization.
  • Encodes priors on change point locations as sample weights, works with linear, logistic, Huber, and Poisson losses.
  • Beats MOSEG, McScan, DPDU, DCDP, charcoal, and GLMBSA on synthetic benchmarks across multiple regimes.
  • Provides precise asymptotic theory for Gaussian designs and an efficiently computable posterior over change points.
  • Released as open-source Python and R packages.
  • Applied to myocardial infarction data and FRED-MD macroeconomic series, recovering known regime changes.

Real-world datasets rarely stay still. A model predicting economic output can shift when policy changes, a medical risk model can behave differently for patients past a certain age, and a linear regression fit across time can hide multiple regime changes. Researchers from Harvard and Cambridge have published Weighted ERM, a method that finds these breakpoints in high-dimensional regression, and released it as ready-to-use Python and R packages.

The paper studies change point estimation in high-dimensional generalized linear models, where you are given data pairs and want to find the sample indices at which the underlying regression vector changes. The setup covers linear, logistic, quantile, and Poisson regression under one umbrella.

Where existing methods leave signal on the table

Change point detection in high-dimensional regression has a rich literature, but almost all of it makes structural assumptions on the coefficients rather than on where the breaks happen. Existing procedures can incorporate assumptions such as sparsity or sparse differences between regression vectors, but they are not equipped to exploit prior information on change point locations.

That gap matters in practice. An analyst often has a rough sense of when a break might occur, say that a regime change is more likely somewhere in the middle third of a time series than near the endpoints. The AMP-based estimator from a recent related work can incorporate priors on both signals and change points, but it is difficult to tailor when the signal prior isn't known, and its guarantees rely on i.i.d. Gaussian design assumptions.