Comparing pronoun resolution algorithms

This paper discusses the comparative evaluation of five well-known pronoun resolution algorithms conducted with the help of a purpose-built tool for consistent evaluation in anaphora resolution, termed the evaluation workbench. The workbench enables the evaluation and comparison of pronoun resolution algorithms on the basis of the same preprocessing tools and test data. The tool is controlled by the user who can conduct the evaluation according to a variety of parameters, with regard to the types of anaphors and the samples used for evaluation. The extensive comparative evaluation of the pronoun resolution algorithms showed that their performance was significantly lower than the figures reported in the original papers describing the algorithms. The evaluation study concluded that the main reason for this drop in performance is the fact that all algorithms operate in a fully automatic mode.