Georgia Tech Proves Anonymized Recommendation Letters Still Leak Gender to AI
A new interpretability case study shows language models can still guess applicant gender from recommendation letters even after names and pronouns are stripped.
Takeaways
- Classifiers guessed applicant gender in medical residency letters at 63% accuracy after pronouns and names were removed.
- DistilBERT, RoBERTa, and Llama 2 all landed near 55-58% macro F1 on the de-gendered corpus of 8,992 letters.
- SHAP and TF-IDF flagged "humanitarian," "delightful," and "children" as female cues; "military," "humble," and "leadership" as male.
- Masking top implicit tokens only dropped DistilBERT F1 by 2.7%, still well above the 50% chance line.
- Prediction-flip analysis showed removing "leadership" flipped male predictions to female three times more often than the reverse.
- Study argues token blocklists cannot fully de-gender evaluative prose; upstream auditing must complement model-level fairness fixes.
Anonymizing a letter of recommendation feels like a solved problem: strip the names, swap the pronouns, and you have a gender-neutral document. A new interpretability case study on nearly 9,000 medical residency letters shows that assumption is wrong. Even after aggressive de-gendering, transformer classifiers still guessed applicant gender well above chance, and the words giving the game away were the ones evaluators use every day.
The paper from researchers at Georgia Tech, Michigan, Northwestern, Purdue, and Iowa State frames this as a bias audit problem for real admissions pipelines rather than a synthetic benchmark. The findings sit uncomfortably close to how many hiring and admissions teams already handle blinding.
The blinding problem nobody wants to talk about
Letters of recommendation shape admissions and hiring at almost every professional gate, and the standard fairness playbook is to remove explicit identifiers. Prior work has shown the limits of scrubbing names and pronouns as a blinding strategy, because implicit gendered language pervades evaluative prose. The open question is how much signal actually survives that scrubbing, and whether the rest can be stripped automatically.
The team assembled a corpus of 8,992 recommendation letters written on behalf of candidates applying to a major U.S. anesthesiology residency program, with 2,787 letters for female applicants and 6,205 for male applicants (roughly 31% female and 69% male). Applicant gender came from self-reported metadata attached to the letters, and names had already been replaced with placeholder tokens like FIRST_NAME before analysis.
How the de-gendering pipeline works
Rather than swap gendered words for neutral placeholders like "they," which tends to mangle grammar, the authors converted every male-coded term to its female counterpart. For coverage, they used regular expressions to capture variations of each term, including plural forms, verb tenses, contractions, punctuation, and positioning within a sentence. That allowed detection and replacement of forms like "she's", "husband.", and "mothers!" with high precision. The gendered term list was aggregated from Bias-BERT and GN-GloVe and included non-obvious markers like "husband," "fraternity," and "sorority."