Skip to main content
Back

Auto-FP Beats TPOT as Evolution Crushes Fancy AI Search Methods

A benchmark of 15 search algorithms across 45 tabular datasets reveals that evolution beats fancy methods, and random search embarrasses most of them

  • First systematic benchmark of automated feature preprocessing, comparing 15 search algorithms across 45 public datasets.
  • Auto-FP is framed as either HPO or NAS, letting existing search algorithms transfer directly.
  • Evolution-based methods, especially PBT, take the top overall ranking across LR, XGB, and MLP downstream models.
  • Random search beats most surrogate-model, RL-based, and bandit-based algorithms including Hyperband and BOHB.
  • Model training dominates runtime, so smarter data sampling matters more than smarter search.
  • Full code and results released at github.com/AutoFP/Auto-FP; Auto-FP beats TPOT's built-in FP module on most datasets.

Picking the right chain of feature preprocessors, things like StandardScaler, PowerTransformer, and MinMaxScaler, still matters a lot for classical machine learning on tabular data. Yet unlike hyperparameter tuning or neural architecture search, almost nobody has systematically studied how to automate it. A team from Simon Fraser University and ETH Zürich just did, and their experimental study of Auto-FP lands on some uncomfortable conclusions for the AutoML community.

Why the pipeline you pick actually matters

Classical models like linear regression and tree-based methods are sensitive to data distribution, so feature preprocessing, which transforms features from one distribution to another, is a crucial step for model quality. A data scientist has to decide both which preprocessors to include and in what order to stack them, and the answer depends on the dataset and the downstream model.

To show this isn't hypothetical, the authors enumerated 2,800 pipelines of length up to four on several datasets. On the Pd dataset, the best pipeline reached 0.94 accuracy while the worst hit 0.24, and running no pipeline at all topped out at 0.81. A bad choice can actively hurt you.

Distribution of logistic regression accuracy across thousands of preprocessing pipelines on four datasets

Borrowing tricks from HPO and NAS

Brute-forcing every ordered combination is hopeless, because with n preprocessors the search space blows up to roughly n^n. The paper's central move is to recognize that Auto-FP can be modelled as either a hyperparameter optimization (HPO) problem or a neural architecture search (NAS) problem, which lets the authors borrow existing search algorithms wholesale. They benchmark 15 of them, grouped into five families: