Chapter 2: Topic Detection and Tracking: Event Based Information Organisation (2002)
Here we define some important terms:
- Topic: A seminal event or activity, along with all directly related events and activities.
- Story: A story is on topic if it discusses events or activities that are directly connected to the topic's seminal event.
The next section talks about the evaluation metrics that are used in order to test the system that is built. Each of the 5 tasks are evaluated as a detection task. For each task there is some input data, a hypothesis and then the system needs to decide whether it is true.
For each task we can consider binary decisions that is it is a target or a non - target. The terminology that we use to evaluate the system is as follows:
- If the reference annotation and the system response agree (that is either both say it is a target or both say it is not a target) then we say we are correct.
- If the system response not a target but we have an annotation saying it is a target then we have a 'missed detection'
- If the system response is a target however the annotation says not a target we say we have a 'false alarm'