Skip to main content
Back

SeClaw Catches Dangerous AI Agent Behavior Before It Reaches Your Users

SeClaw auto-generates security test tasks from structured risk specs and evaluates agents inside Docker sandboxes, tracking every tool call along the way

  • SeClaw is a new framework for auto-generating security test tasks for LLM agents from structured risk specs, then evaluating agents inside isolated Docker sandboxes.
  • Unlike existing benchmarks, it records full execution trajectories -- every tool call, file operation, and intermediate step -- not just the final model output.
  • Tasks are synthesized by a multi-agent pipeline that expands structured risk specifications into executable scenarios, enabling continuous updates as new threats emerge.
  • Coverage spans four risk categories: resource risks, task risks, environment risks, and intrinsic agent behaviors.
  • The framework uses ToolHub and MCP (Model Context Protocol) to reconstruct realistic tool environments for each test scenario.
  • The paper is marked preliminary; code is open-source on GitHub with plans to expand to Claude Code and other agent harnesses.

Autonomous LLM agents increasingly operate in stateful environments where they access tools, files, memory, and external services -- and while those capabilities unlock powerful real-world workflows, they also introduce security risks that are genuinely hard to measure. The field has been scrambling to build benchmarks fast enough to keep up, but the existing ones have a structural problem. Current agent security benchmarks often rely on manually curated tasks, provide limited coverage of emerging threats, and focus primarily on final outcomes rather than the execution processes that lead to unsafe behavior.

The gap existing benchmarks leave open

Think about what it means to evaluate a security-sensitive agent. You don't just want to know whether it produced a bad final answer -- you want to know when it made a dangerous tool call, which file it accessed, and how a benign-looking user task escalated into something unsafe mid-trajectory. Security incidents involving LLMs are increasingly tied to emergent behavior rather than discrete vulnerabilities. Attackers are not just exploiting bugs; they are manipulating how models interpret instructions, assemble context, and interact with connected tools. A benchmark that only checks the last line of output misses all of that.

The manual curation problem is equally painful. Writing realistic security tasks by hand is slow, expensive, and produces a fixed dataset that goes stale the moment a new threat pattern emerges. What the field needs is a way to generate security tasks systematically, from a living description of what risks look like -- and then run those tasks in an environment that captures every step of agent behavior.

What SeClaw actually does

SeClaw is a framework that combines specification-driven security task synthesis with execution-based security evaluation for autonomous agents. Spec-driven security task synthesis enables scalable and controllable construction of security tasks from structured risk specifications, while SeClaw Docker provides a standardized testbed for evaluating agent behavior under diverse safety-risk scenarios.

The framework is split into two stages that feed into each other: