You are here

Challenges in evaluating summaries of short stories

TitleChallenges in evaluating summaries of short stories
Publication TypeConference Paper
Year of Publication2006
AuthorsKazantseva, A, Szpakowicz, S
Conference NameWorkshop on Task-Focused Summarization and Question Answering
PublisherAssociation for Computational Linguistics
Conference LocationSydney, Australia
ISBN Number1-932432-79-5
Keywordsautomatic summarization

This paper presents experiments with the evaluation of automatically produced summaries of literary short stories. The summaries are tailored to a particular purpose of helping a reader decide whether she wants to read the story. The evaluation procedure includes extrinsic and intrinsic measures, as well as subjective and factual judgments about the summaries pronounced by human subjects. The experiments confirm the experience of summarizing more conventional genres: sentence overlap between human- and machine-made summaries is not a complete picture of the quality of a summary. In fact, in our case, sentence overlap does not correlate well with human judgment. We explain the evaluation procedures and discuss several challenges of evaluating summaries of works of fiction.