Measuring Game Engagement

Multiple Methods and Construct Complexity

Rosa Mikeal Martey, Kate Kenski, James Folkestad, Laurie Feldman, Elana Gordis, Adrienne Shaw, Jennifer Stromer-Galley, Ben Clegg, Hui Zhang, Nissim Kaufman, Ari N. Rabkin, Samira Shaikh, Tomek Strzalkowski

Research output: Contribution to journalArticle

16 Citations (Scopus)

Abstract

Background. Engagement has been identified as a crucial component of learning in games research. However, the conceptualization and operationalization of engagement vary widely in the literature. Many valuable approaches illuminate ways in which presence, flow, arousal, participation, and other concepts constitute or contribute to engagement. However, few studies examine multiple conceptualizations of engagement in the same project.

Method. This article discusses the results of two experiments that measure engagement in five different ways: survey self-report, content analyses of player videos, electro-dermal activity, mouse movements, and game click logs. We examine the relationships among these measures and assess how they are affected by the technical characteristics of a 30-minute, custom-built, educational game: use of a customized character, level of narrative complexity, and level of art complexity.

Results. We found that the five measures of engagement correlated in limited ways, and that they revealed substantially different relationships with game characteristics. We conclude that engagement as a construct is more complex than is captured in any of these measures individually and that using multiple methods to assess engagement can illuminate aspects of engagement not detectable by a single method of measurement.

Original languageEnglish (US)
Pages (from-to)528-547
Number of pages20
JournalSimulation and Gaming
Volume45
DOIs
StatePublished - Jan 24 2014

Fingerprint

Experiments
Conceptualization

Keywords

  • art
  • art complexity
  • attention
  • character customization
  • content analyses
  • customized character
  • EDA
  • educational games
  • electro-dermal activity
  • engagement
  • game clicks
  • interactivity
  • measurement
  • mouse movements
  • multiple methods
  • narrative
  • narrative complexity
  • presence
  • self-report
  • transportation

ASJC Scopus subject areas

  • Computer Science Applications
  • Business, Management and Accounting (miscellaneous)

Cite this

Martey, R. M., Kenski, K., Folkestad, J., Feldman, L., Gordis, E., Shaw, A., ... Strzalkowski, T. (2014). Measuring Game Engagement: Multiple Methods and Construct Complexity. Simulation and Gaming, 45, 528-547. https://doi.org/10.1177/1046878114553575

Measuring Game Engagement : Multiple Methods and Construct Complexity. / Martey, Rosa Mikeal; Kenski, Kate; Folkestad, James; Feldman, Laurie; Gordis, Elana; Shaw, Adrienne; Stromer-Galley, Jennifer; Clegg, Ben; Zhang, Hui; Kaufman, Nissim; Rabkin, Ari N.; Shaikh, Samira; Strzalkowski, Tomek.

In: Simulation and Gaming, Vol. 45, 24.01.2014, p. 528-547.

Research output: Contribution to journalArticle

Martey, RM, Kenski, K, Folkestad, J, Feldman, L, Gordis, E, Shaw, A, Stromer-Galley, J, Clegg, B, Zhang, H, Kaufman, N, Rabkin, AN, Shaikh, S & Strzalkowski, T 2014, 'Measuring Game Engagement: Multiple Methods and Construct Complexity', Simulation and Gaming, vol. 45, pp. 528-547. https://doi.org/10.1177/1046878114553575
Martey RM, Kenski K, Folkestad J, Feldman L, Gordis E, Shaw A et al. Measuring Game Engagement: Multiple Methods and Construct Complexity. Simulation and Gaming. 2014 Jan 24;45:528-547. https://doi.org/10.1177/1046878114553575
Martey, Rosa Mikeal ; Kenski, Kate ; Folkestad, James ; Feldman, Laurie ; Gordis, Elana ; Shaw, Adrienne ; Stromer-Galley, Jennifer ; Clegg, Ben ; Zhang, Hui ; Kaufman, Nissim ; Rabkin, Ari N. ; Shaikh, Samira ; Strzalkowski, Tomek. / Measuring Game Engagement : Multiple Methods and Construct Complexity. In: Simulation and Gaming. 2014 ; Vol. 45. pp. 528-547.
@article{3ed0c57ca8b54a3483f9b2263cf6f409,
title = "Measuring Game Engagement: Multiple Methods and Construct Complexity",
abstract = "Background. Engagement has been identified as a crucial component of learning in games research. However, the conceptualization and operationalization of engagement vary widely in the literature. Many valuable approaches illuminate ways in which presence, flow, arousal, participation, and other concepts constitute or contribute to engagement. However, few studies examine multiple conceptualizations of engagement in the same project.Method. This article discusses the results of two experiments that measure engagement in five different ways: survey self-report, content analyses of player videos, electro-dermal activity, mouse movements, and game click logs. We examine the relationships among these measures and assess how they are affected by the technical characteristics of a 30-minute, custom-built, educational game: use of a customized character, level of narrative complexity, and level of art complexity.Results. We found that the five measures of engagement correlated in limited ways, and that they revealed substantially different relationships with game characteristics. We conclude that engagement as a construct is more complex than is captured in any of these measures individually and that using multiple methods to assess engagement can illuminate aspects of engagement not detectable by a single method of measurement.",
keywords = "art, art complexity, attention, character customization, content analyses, customized character, EDA, educational games, electro-dermal activity, engagement, game clicks, interactivity, measurement, mouse movements, multiple methods, narrative, narrative complexity, presence, self-report, transportation",
author = "Martey, {Rosa Mikeal} and Kate Kenski and James Folkestad and Laurie Feldman and Elana Gordis and Adrienne Shaw and Jennifer Stromer-Galley and Ben Clegg and Hui Zhang and Nissim Kaufman and Rabkin, {Ari N.} and Samira Shaikh and Tomek Strzalkowski",
year = "2014",
month = "1",
day = "24",
doi = "10.1177/1046878114553575",
language = "English (US)",
volume = "45",
pages = "528--547",
journal = "Simulation and Gaming",
issn = "1046-8781",
publisher = "SAGE Publications Inc.",

}

TY - JOUR

T1 - Measuring Game Engagement

T2 - Multiple Methods and Construct Complexity

AU - Martey, Rosa Mikeal

AU - Kenski, Kate

AU - Folkestad, James

AU - Feldman, Laurie

AU - Gordis, Elana

AU - Shaw, Adrienne

AU - Stromer-Galley, Jennifer

AU - Clegg, Ben

AU - Zhang, Hui

AU - Kaufman, Nissim

AU - Rabkin, Ari N.

AU - Shaikh, Samira

AU - Strzalkowski, Tomek

PY - 2014/1/24

Y1 - 2014/1/24

N2 - Background. Engagement has been identified as a crucial component of learning in games research. However, the conceptualization and operationalization of engagement vary widely in the literature. Many valuable approaches illuminate ways in which presence, flow, arousal, participation, and other concepts constitute or contribute to engagement. However, few studies examine multiple conceptualizations of engagement in the same project.Method. This article discusses the results of two experiments that measure engagement in five different ways: survey self-report, content analyses of player videos, electro-dermal activity, mouse movements, and game click logs. We examine the relationships among these measures and assess how they are affected by the technical characteristics of a 30-minute, custom-built, educational game: use of a customized character, level of narrative complexity, and level of art complexity.Results. We found that the five measures of engagement correlated in limited ways, and that they revealed substantially different relationships with game characteristics. We conclude that engagement as a construct is more complex than is captured in any of these measures individually and that using multiple methods to assess engagement can illuminate aspects of engagement not detectable by a single method of measurement.

AB - Background. Engagement has been identified as a crucial component of learning in games research. However, the conceptualization and operationalization of engagement vary widely in the literature. Many valuable approaches illuminate ways in which presence, flow, arousal, participation, and other concepts constitute or contribute to engagement. However, few studies examine multiple conceptualizations of engagement in the same project.Method. This article discusses the results of two experiments that measure engagement in five different ways: survey self-report, content analyses of player videos, electro-dermal activity, mouse movements, and game click logs. We examine the relationships among these measures and assess how they are affected by the technical characteristics of a 30-minute, custom-built, educational game: use of a customized character, level of narrative complexity, and level of art complexity.Results. We found that the five measures of engagement correlated in limited ways, and that they revealed substantially different relationships with game characteristics. We conclude that engagement as a construct is more complex than is captured in any of these measures individually and that using multiple methods to assess engagement can illuminate aspects of engagement not detectable by a single method of measurement.

KW - art

KW - art complexity

KW - attention

KW - character customization

KW - content analyses

KW - customized character

KW - EDA

KW - educational games

KW - electro-dermal activity

KW - engagement

KW - game clicks

KW - interactivity

KW - measurement

KW - mouse movements

KW - multiple methods

KW - narrative

KW - narrative complexity

KW - presence

KW - self-report

KW - transportation

UR - http://www.scopus.com/inward/record.url?scp=84921455575&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84921455575&partnerID=8YFLogxK

U2 - 10.1177/1046878114553575

DO - 10.1177/1046878114553575

M3 - Article

VL - 45

SP - 528

EP - 547

JO - Simulation and Gaming

JF - Simulation and Gaming

SN - 1046-8781

ER -