Mark a retrieved item as relevant or nonrelevant by clicking a number on either side and watch how measures change. Mouseover the measures to see their definitions.

In order to evaluate and compare information retrieval systems, traditionally one uses a "test collection" composed of three entities: • a set of queries (or "information needs") • a target dataset (where to look for items that will satisfy the information needs) • a set of relevance assessments (the items that satisfy the information needs)

This test collection is generated artificially and it is assumed to be available.

Based on this test collection one can compare the results of a query using two different information retrieval systems. The items retrieved by the two systems are represented by the two sequences of circles on the left and right.

There are two ways to interact with this Demonstration: 1. by using the controls at the top 2. by marking items as relevant/nonrelevant in the graphics

Use the controls to set the number of relevant and nonrelevant items for the query being considered. These numbers are characteristics of the test collection for each given query.

Relevant and nonrelevant items do not necessarily sum up to all the items in the collection, but they are those items in the collection that are known to be relevant or not to a given query because their relevance has been assessed (or inferred).

The main concern of information retrieval evaluation is to decide the better retrieval system based on a test collection. Is the system that retrieves the most relevant items the better one? Or the one with the higher average precision? Or the one with higher normalized cumulative gain?