Topic Dectection and Tracking Evaluation Overview

Johnathan G. Fiscuss and George R. Doddington
Chapter 2: Topic Detection and Tracking: Event Based Information Organisation (2002)

Here we define some important terms:

• Topic: A seminal event or activity, along with all directly related events and activities.
  
• Story: A story is on topic if it discusses events or activities that are directly connected to the topic's seminal event.
  

There does need to be some limit to the time that we consider the story to be on topic.

The next section talks about the evaluation metrics that are used in order to test the system that is built. Each of the 5 tasks are evaluated as a detection task. For each task there is some input data, a hypothesis and then the system needs to decide whether it is true.

For each task we can consider binary decisions that is it is a target or a non - target. The terminology that we use to evaluate the system is as follows:

• If the reference annotation and the system response agree (that is either both say it is a target or both say it is not a target) then we say we are correct.
• If the system response not a target but we have an annotation saying it is a target then we have a 'missed detection'
• If the system response is a target however the annotation says not a target we say we have a 'false alarm'

The remainder of the chapter again deals with the 5 tasks and a more detailed description of the tasks. Further is mentions some assumptions that we make. We shall revisit this when we look at the tasks that we are performing.