OpenEvolve Rewrites PassLLM's Prompt and Cracks 4x More Passwords
Automated prompt evolution boosts LLM password cracking rates from 2.02% to 8.48%, exposing how small text tweaks can quadruple attacker success.
Takeaways
- Prompt-only evolution lifts PassLLM cracking rate from 2.02% to 8.48% on a RockYou holdout set.
- Pipeline combines PassLLM with OpenEvolve using MAP-Elites and a three-island population with 10% migration.
- Local Qwen3 8B on one 24GB GPU reaches 8.28%, essentially matching a frontier cloud ensemble at 8.48%.
- Large gains often appear within the first 1 to 10 evolutionary iterations before the archive stabilizes.
- Per-symbol F-score AUC improves 26%, showing evolved prompts model realistic character distributions rather than just common tokens.
- Results directly refute PassLLM's original claim that prompt content has minimal impact on cracking performance.
A short instruction line at the top of a prompt turns out to be worth roughly a 4x jump in password cracking performance. A new study on LLM-guided prompt evolution takes an existing LLM-based password guesser, leaves its weights untouched, and rewrites the system prompt using an evolutionary search loop. Cracking rate climbs from 2.02% to 8.48% on a RockYou holdout set, and a local 8B model on a single 24GB GPU gets within a fraction of a percentage point of a frontier cloud ensemble.
Why passwords are still the softest target
Password guessing has evolved from dictionary attacks and rule engines like John the Ripper and Hashcat, to probabilistic context-free grammars (PCFGs), to Markov chains, to RNNs and GANs such as PassGAN. More recent Transformer-based approaches like PassBERT use bi-directional models with a pre-training/fine-tuning paradigm to improve real-world attacks, including conditional, targeted, and adaptive rule-based guessing. The current state of the art, PassLLM, fine-tunes small Llama, Mistral and Qwen models on leaked corpora using LoRA and outperforms all of these.
PassLLM has one quirk. Its authors reported that the wording of the inference prompt barely matters, and this new paper directly refutes that claim. The baseline prompt they inherited is a single sentence asking the model to act as a trawling password guessing model, and it leaves a lot of performance on the floor.
Evolve the prompt, not the weights
The pipeline glues PassLLM (built on Qwen2.5-0.5B-Instruct with a LoRA adapter) to