Wayfarer learns long-horizon Atari skills from pixels

University of Alberta researchers have introduced Wayfarer, an online reinforcement-learning agent that discovers temporally extended skills, known as options, from pixel observations. The arXiv preprint reports the strongest aggregate results among the systems evaluated on 10 Atari 2600 games selected for difficult exploration and long-term credit assignment.

Wayfarer learns which parts of an observation the agent can control, extracts large-scale structure from that representation, and converts the resulting features into reusable behaviors. A high-level policy can then choose either a primitive action or a learned option, reducing the number of decisions required to traverse long sequences.

Why flat control stalls

Conventional Atari agents choose among primitive actions such as moving left, moving right, or pressing fire. That approach works well when rewards follow actions quickly. Sparse-reward games create a harder problem because the agent may need to execute hundreds of correct actions before receiving evidence that any earlier decision helped.

An option packages a sequence of actions into a mini-policy with its own initiation and termination conditions. An option might move a character to a ladder and climb down, allowing the control policy to select that complete behavior instead of issuing each directional command separately. This temporal abstraction can accelerate exploration and shorten the chain of decisions between an action and its eventual reward.

Practical option discovery has remained difficult. Earlier methods often relied on small state spaces, handcrafted features, or structured inputs such as emulator RAM. Classical eigenoption research from the same lab used singular value decomposition over compact state features, an approach that does not directly scale to high-dimensional pixel observations.