Human Resources
Interested in simulations? Don’t miss the big (scoring) reveal!


Interested in simulations? Don’t miss the big (scoring) reveal!


Loading

Mar 21, 2017 | by Eleni Lobene & Bharati Belwalkar


Simulations are one of the most robust assessment methodologies available on the market today due to their ability to predict important work outcomes1,4 and their ability to replicate high-fidelity work experiences1. An effective simulation:

These requirements cannot be fulfilled unless the simulation is designed with special consideration to its scoring approaches. However, scoring candidate performance on simulations has been sporadically discussed in the field of Industrial and Organizational psychology. To our knowledge, very few resources extensively discuss this subject,2,3 leaving limited guidance on scoring simulations.

Scoring a simulation begins with identifying the purpose (e.g., hiring, training), how the scores will be used (e.g., as cut scores vs. hurdles), and the type of simulation in question (e.g., virtual interviewing, ‘gamified’ assessments)3.  Computing scores at more granular levels and combining data points in multiple, innovative ways can yield much needed flexibility when designing both custom and off-the-shelf simulations.

Recognizing that the range of simulation scoring approaches and options are virtually limitless, we are hosting an original session at the 32nd Annual Conference of the Society for Industrial and Organizational Psychology to initiate a conversation on simulation scoring to uncover current specific practices in the industry, identify differences in approaches and opinions, and inspire discussion regarding future advancements. In order to obtain multiple perspectives, we have recruited panelists who design simulation assessments on a daily basis. This expert panel is comprised of five professionals who are a balanced mix of early career and seasoned consultants, and will offer perspectives on various scoring approaches used by their organizations.

A dynamic group of practitioners, they will share some of their most innovative simulation scoring practices in this engaging and interactive session. The topics of these presentations include:

  • Scoring short- and long-form raw text that is entered into a simulation
  • Balancing psychometric rigor and innovation
  • Ensuring accurate competency measurement
  • Leveraging behavioral data beyond time and accuracy
  • Comparing alternative scoring approaches for situational judgment tests

Their presentations will be followed by a panel discussion focused on case studies and empirical evidences of diverse and complex scoring practices. If you are curious to know the expert panelists’ responses to the following questions or would like to pose your own burning question, you will not want to miss this session:

1. What makes scoring in simulation settings different than scoring of other types of assessment?

2. What is the biggest unresolved challenge related to scoring you face in your organization?

3. Handling qualitative and quantitative data requires different processes and skill sets. What best practices can you share about leveraging these two types of data?

References:

1.Lievens, F., & Patterson, F. (2011). The Validity and Incremental Validity of Knowledge Tests, Low-Fidelity Simulations, and High-Fidelity Simulations for Predicting Job Performance in Advanced-Level High-Stakes Selection. Journal of Applied Psychology, 96, 927-940.

2.
Salas, E., Rosen, M. A., Held, J. D., & Weissmuller, J. J. (2009). Performance measurement in simulation-based training a review and best practices. Simulation & Gaming, 40(3), 328-376.

3. Sydell, E., Ferrell, J., Carpenter, J., Frost, C., & Brodbeck, C. C. (2013). Simulation scoring. In Simulations for Personnel Selection (pp. 83-107). Springer New York.

4. Thornton, G. C., & Cleveland, J. N. (1990). Developing managerial talent through simulation. American Psychologist, 45(2), 190-199.

Want to learn more about our Leadership & Assessment solutions?

About the Author