Eye Tracking Analysis of User Behavior in WWW Search

Laura Granka Thorsten Joachims Geri Gay

why use eye-tracking for information retrieval?

Understand how searchers evaluate online search results Enhanced interface design More accurate interpretation of implicit feedback (eg, clickthrough data) More targeted metrics for evaluating retrieval performance
Figure: popular regions are highlighted through shadow-intensity

key research questions

How long does it take searchers to select a document? How many abstracts do searchers look at before making a selection? Do searchers look at abstracts ranked lower than the selected document? Do searchers view abstracts linearly? Which parts of the abstract are most likely to be viewed?

what is eye-tracking?
Device to detect and record where and what people look at Multiple applications: reading, usability, visual search, in both physical and virtual contexts
Figure: Cornell HCI eye-tracking configuration

Eye tracking device

View of subjects pupil on monitor; used for calibration

ocular indices for www tracking

Fixations: ~200-300ms; information is acquired Saccades: extremely rapid movements between fixations Pupil dilation: size of pupil indicates interest, arousal

Aggregate eye-tracking graphs depict viewing intensity in key regions

Scanpath output depicts pattern of movement throughout screen. Black markers represent fixations.

experimental search tasks

Ten search tasks given to all participants Search topics included travel, science, movies, local, television, college, and trivia Searches evenly split between informational and navigational tasks

experimental procedures
Users conducted live Google searches Users allowed to search freely, with any queries Script removed all ad content Proxy stored all pages and log files

Figure: Specific zones were created around each result, enabling eye-movements to be analyzed specific to the rankings

sample eye-tracking output

sample eye-tracking output

sample eye-tracking output

sample eye-tracking output

sample eye-tracking output

overall searching behavior

How long does it take users to select a document?
Tim e to select docum ent 10 9 8 7 6 5 4 3 2 1 0

Time spent before a result is clicked

mean time (s)

du de ra nc h ti m em ac hi ne m ou nt ai n

jo rd an

ho us in g

co rn el l

pr im ar y

more difficult

task

Overall mean: 5.7 seconds, St.D: 5.4

gr ey ho un d

an ti b io t ic s

em er il

less difficult

overall viewing behavior

How many abstracts do we view, and in what order? Most likely to view only two documents per results set
Total number of abstracts viewed per page
120 100

frequency

80 60 40 20 0 1 2 3 4 5 6 7 8 9 10

Notice dip after page break

Total number of abstracts viewed

Mean: 3.07

Median/Mode: 2.00

overall viewing behavior

How many abstracts do we view, and in what order? Results viewed linearly
Instance of arrival to each result
mean fixation value of arrival
25 20 15 10 5 0 1 2 3 4 5 6 7 8 9 10 Rank of result

overall viewing behavior

spent each result by viewing frequency ofeach doc selected How much Time time doinwe spend abstract?
180 160

# times result selected time spent in abstract

1 0.9 0.8 0.7 0.6 0.5

# times rank selected

140 120 100 80 60 40 20 0 1 2 3 4 5 6 7 8 9 10 11

0.4 0.3 0.2 0.1 0

Rank of result

Time spent viewing each abstract compared with the frequency that each rank is selected. Error bars are 1 SE

mean time (s)

overall viewing behavior

How thoroughly do we view the results set?
Number of abstracts viewed above and below selected link

overall viewing behavior

What information in the abstract is most useful?
Percentage of time spent viewing each part of abstract

Title: 30% Snippet: 43% Category: 0.3% URL: 21%

Other: 5% (includes,
cached, similar pages, description) URL: 21.1%

Other: 5.3% Title: 30.5%

Category: 0.3% Snippet: 42.8%

overall searching behavior

Search task difficulty and satisfaction with Google
Difficulty
9 8 7 6

Satisfaction

ranking

5 4 3 2 1 0
Cornell mansion M ichael Jordan Dude ranch Time M achine Tallest mountain CM U housing NY Primary First antibiotic Emeril Greyhound

search task

more difficult

less difficult

*Difficulty and satisfaction are ranked on a 1-10 scale; 10 meaning very difficult and very satisfied, respectively

overall searching behavior

Task difficulty
influences rank of selected document and number of abstracts viewed
Difficulty
9 8 7 6

Satisfaction

ranking

5 4 3 2 1 0
Cornell mansion M ichael Jordan Dude ranch Time M achine Tallest mountain CM U housing NY Primary First antibiotic Emeril Greyhound

search task

more difficult

less difficult

Mean rank of selected doc: 2.66 Median/ Mode: 1.00

overall searching behavior

Top Query Terms
1. Michael Jordan statistician
Frequency

20

2. Thousand acres dude ranch

2. One thousand acres dude ranch 3. 1000 acres dude ranch 4. Time machine movie

11
11 9 7

4. Carnegie mellon university graduate housing

5. Imdb 5. Emeril lagasse 5. First modern antibiotic 5. Greyhound bus 5. Carnegie mellon graduate housing

7
6 6 6 6 6

conclusions
Searching Trends: Popularity of specialized, vertical portals
Majority of students preferred an internal imdb.com search over a general Google search

Several students preferred conducting a Google search from the cmu.edu homepage

conclusions
Document selected in under 5 seconds Users click on the first promising link they see Results viewed linearly Top 2 results most likely to be viewed Users rather reformulate query than scroll Task type and difficulty affect viewing behavior Presentation of results affects selection

future research
Impact on advertising
With such fast selections being made, will searchers even view ads?

Ads most likely to be seen: Difficult task Ambiguous info need Informational query Low searcher expertise

future research
Relevance judgments
Do we spent more time viewing relevant abstracts? Do we click the first relevant abstract viewed? Does pupil dilation increase for more relevant documents

If results were re-ranked, would viewing behavior differ?

Cornell University Computer Science & Human-Computer Interaction

Thorsten Joachims tj@cs.cornell.edu Laura Granka granka@cornell.edu Matthew Feusner feusner@cornell.edu Geri Gay gkg1@cornell.edu Helene Hembrooke hah4@cornell.edu