Jadavpur University Finds Fine-Tuned PEGASUS Beats GPT-3.5 at Writing Research Titles
A fine-tuned PEGASUS model beats GPT-3.5 and LLaMA-3-8B at generating research paper titles from abstracts, with a new social sciences dataset to boot
- PEGASUS wins: Fine-tuned PEGASUS-large beats GPT-3.5-turbo and LLaMA-3-8B at generating research paper titles from abstracts across most metrics.
- New dataset: The paper introduces SpringerSSAT, a new dataset of abstract-title pairs from four Springer social science journals.
- Smaller + fine-tuned > larger + zero-shot: Domain-specific fine-tuning on a compact model consistently outperforms large models used out of the box.
- Creative titles: ChatGPT generates more creative titles even when it loses on accuracy metrics, making it a complementary tool.
- Cross-domain generalization: PEGASUS fine-tuned on CS papers still achieves top performance on NLP conference papers without retraining.
- Live demo available: A web app lets users paste an abstract and generate titles using the fine-tuned models.
Picking a title for a research paper is deceptively hard. It must succinctly convey the main contribution of a paper while remaining brief and appealing to a broad audience. Most researchers spend hours agonizing over it. A new paper published in the International Journal on Digital Libraries asks: what if a language model could do it for you?
Automatic title generation can be viewed as a form of abstractive text summarization, which aims to capture and represent the most salient aspects of a document in a compact form. The team from Jadavpur University and the Indian Association for the Cultivation of Science framed the problem exactly that way, and ran a systematic comparison of models ranging from compact fine-tuned PLMs (pre-trained language models) to GPT-3.5-turbo.
The Bigger Models Don't Always Win
Fine-tuned PEGASUS-large outperforms other models, including fine-tuned LLaMA-3-8B and zero-shot GPT-3.5-turbo, across most metrics. That's a striking result. PEGASUS is orders of magnitude smaller than either of those two, yet it consistently produces titles closer to what human authors actually write.
The reason comes down to how PEGASUS was originally trained. In PEGASUS, important sentences are removed and masked from an input document and are generated together as one output sequence from the remaining sentences, similar to an extractive summary. That pre-training objective is essentially a rehearsal for exactly this task: compress a document into its most important sentence. Fine-tuning on abstract-title pairs then sharpens that skill further.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
