Queen Mary's BiCon-Gate Fixes Messy Chat Claims Before AI Fact-Checkers Break
A gated rewrite pipeline uses bidirectional NLI to decide when to trust a de-colloquialised claim, improving DialFact retrieval and verification.
Takeaways
- BiCon-Gate uses bidirectional NLI plus cosine similarity to route dialogue claims between a rewrite and the original.
- Staged pipeline handles contractions, punctuation, true-casing, and scoped pronoun resolution using Maverick and Llama-3.1-8B.
- On DialFact, R4+Gated hits 62.93% FV macro-F1, a +1.84 gain over the untouched original claim.
- Retrieval improves across BM25, E5, and BGE stages, with BM25 recall@180 rising from 73.89% to 76.54%.
- A one-shot Qwen2.5-14B rewrite baseline drifts semantically and drops FV macro-F1 to 56.93%.
- Gate accepts scoped rewrites 46.56% of the time but only 0.92% of aggressive one-shot rewrites.
Fact-checking a well-formed sentence from a Wikipedia article is one thing. Fact-checking a chat message like remember when it came out? the public hated it at first! is another entirely. Pronouns dangle, capitalisation is missing, and apostrophes have gone AWOL, all of which quietly wrecks the retrieval and verification pipelines that fact-checkers depend on.
A new paper from Queen Mary University of London tackles this with a small but carefully engineered idea: rewrite the messy claim into something cleaner, but only commit to the rewrite when a semantic consistency check says it is safe. The system is called BiCon-Gate, and it delivers measurable gains on the DialFact benchmark without touching the underlying retriever or verifier.
Why colloquial claims break fact-checkers
Most fact-checking systems follow a retrieve-then-verify pipeline: pull candidate evidence from a knowledge source, then run an entailment model to decide SUPPORTS, REFUTES, or NEI (not enough info). In dialogue, colloquial phenomena such as contractions, missing punctuation, inconsistent casing, ellipsis, and pronominal references can blur claim boundaries and entity mentions, making both retrieval and verification brittle.
The numbers back this up. Kim et al. showed that converting formal FEVER claims into colloquial variants dropped document recall from 90.00% to 72.20%. On DialFact itself, 42.1% of claims contain at least one in-claim pronoun, meaning the verifier is routinely asked to judge sentences whose subject is a bare he, she, or it.
The obvious fix is to rewrite the claim to be self-contained before feeding it downstream, but this creates what the authors call the IR-FV trade-off: aggressive rewriting can boost retrieval recall while introducing semantic drift that misleads verification. Chamoun et al. found that naively substituting in-claim pronouns raised document recall from 56.85% to 67.00% but hurt sentence-level evidence selection and dropped fact-verification accuracy from 44.06% to 42.71%.