ASLPI Research & Statistics

Gallaudet commissioned research studies on the ASLPI to:
Importance of Research:

Defining Validity of an assessment:
Validity refers to the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests. (Standards, pg. 9)

Defining Reliability of an assessment:
Reliability refers to the consistency of measurements when the testing procedure is repeated on a population of individuals or groups.” (Standards, pg. 25)

ASLPI is rater-based assessment. Therefore, the ASLPI interviewers must be skilled and strongly focused on the construct being measured throughout the interview process.

Measurement Structure of the ASL Proficiency Construct as Assessed by the ASLPI


The findings provided evidence that a majority of the variability in the five dimensions was attributable to a single higher-order construct. That is, a majority of the variability in dimension ratings can be meaningfully traced to a single underlying factor, which is as intended by the test developer. Moreover, the relationships between the dimensions and the single underlying factor were strongly consistent (i.e., approximately equal in magnitude). In sum, the five dimensions are representative of a single underlying construct, and the dimensions relate to the underlying construct in a consistent manner—these findings align with the design and current rating protocol of the ASLPI.

The findings also provided evidence that a majority of the variability in the observed ratings was attributable to their respective lower-order construct (i.e., dimension). Ratings were applied consistently and reliably to Grammar, Vocabulary, Fluency, Production/Accent, and Comprehension—indicating that interviewees’ actual proficiency on each dimension was primarily responsible for raters’ judgments of ASLPI performance. Only a small amount of variability in the ratings was attributable to unique, rater-specific variance (or bias). This increases confidence that ASLPI ratings differ as a function of differences in proficiency rather than as a function of the rater characteristics (e.g., harshness/severity, inconsistency) or other sources of error. These findings are supportive of the intended internal structure of the ASLPI and align with the design and current rating protocol of this assessment.

Preliminary Validity Study Conclusion:

Study Sample:
1568 assessment interviews rated by ASLPI evaluators Inter and Intra-rater Reliability - measure of consistency between ratings of the same test taker from different evaluators. This is an ‘intra-class correlation’ (ICC) for a single rater or the average rating across 3 raters.

Correlation Across All Possible Evaluator Pairs
Repeatability of ASLPI Ratings
ASLPI Re-Rating

SWA Consulting, Inc (SWA)
311 S. Harrington St., Suite 200
Raleigh, NC 27603

Dr. Eric A. Surface is the president of SWA Consulting, Inc. (SWA; formerly Surface, Ward, & Associates), a management consulting and applied personnel research firm based in Raleigh, NC. SWA focuses on developing evidence-based human performance solutions for clients. For over a decade, Dr. Surface has worked with military, non-profit, and private-sector organizations-such as the United States Special Operations Command, the American Council on the Teaching of Foreign Languages, and IBM-on projects related to performance, training, work-related foreign language, work analysis, testing, and organizational effectiveness. Dr. Surface has been published in Personnel Psychology, Organizational Research Methods, Human Performance, Military Psychology, Journal of Management, and Foreign Language Annals. He has presented numerous papers at academic, professional, and military conferences and has served as a reviewer for several conferences and journals. His research interests include training effectiveness (broadly defined), the interaction of context and individual differences to influence criteria, foreign language proficiency testing and training, the use of technology for training delivery, test validation, and survey methodology. Dr. Surface earned his PhD in industrial/organizational psychology from NCSU, his MA from East Carolina University, and his BA from Wake Forest University. He is a former Consortium Research Fellow and Consortium Post-Doctoral Research Fellow with the Army Research Institute.