Papers by Patrick Kyllonen
In this symposium, we present the overall design, data, and scientific findings from the ETS Coll... more In this symposium, we present the overall design, data, and scientific findings from the ETS Collaborative Science Assessment Prototype (ECSAP). We are opening our data to the CSCL community and introducing the procedures to request access to the data. ECSAP was developed to explore the assessment of collaborative problem solving (CPS) competency through a large-scale and standardized approach. The goal of this symposium is to examine research questions that are of interest to the CSCL community, such as how CPS skills and collaborative patterns interact with performance outcomes, and how prior content knowledge and personality of team members affect the collaboration process and outcomes. In our study, we collected both individual and collaborative responses (~1500 responses) to the ECSAP instruments. We present our study findings that used new methodologies in psychometrics and followed the best practices of psychometrics and statistics.
Assessing collaborative problem solving (CPS) is an integrated part of the computer-supported col... more Assessing collaborative problem solving (CPS) is an integrated part of the computer-supported collaborative learning (CSCL). We present some preliminary results from a project developed for assessing the CPS using web-based simulation. In the simulation, two participants collaborate via a chat box to complete a task on volcano science. By comparing the responses from 486 individuals and 278 teams (dyads) recruited from Amazon Mechanical Turk, we found the performance from the teams (dyads) is significantly higher than that from individuals. We also find that the item difficulty in the simulation affects both the processes and outcomes of the collaboration.
With the aid of educational data mining and statistical analysis, we investigate the relationship... more With the aid of educational data mining and statistical analysis, we investigate the relationship between collaboration outcomes and collaborative problem solving (CPS) skills exhibited during the collaboration process. We found that negotiation skill contributes positively to the collaboration outcomes while purely sharing information does the opposite.
ETS Research Report Series, 2018
This is the author manuscript accepted for publication and has undergone full peer review but has... more This is the author manuscript accepted for publication and has undergone full peer review but has not been through the copyediting, typesetting, pagination and proofreading process, which may lead to differences between this version and the Version of Record. Please cite this article as
Journal of Intelligence, 2018
Economic inequality has been described as the defining challenge of our time, responsible for a h... more Economic inequality has been described as the defining challenge of our time, responsible for a host of potential negative societal and individual outcomes including reduced opportunity, decreased health and life expectancy, and the destabilization of democracy. Education has been proposed as the “great equalizer” that has and can continue to play a role in reducing inequality. One means by which education does so is through the development of complex problem solving skills in students, skills used to solve novel, ill-defined problems in complex, real-world settings. These are highly valued in the workforce and will likely continue to be so in the future workforce. Their importance is evident in results from employer surveys, as well as by their inclusion in large scale international and domestic comparative assessments. In this paper, I review various definitions of complex problem solving and approaches for measuring it, along with findings from PISA 2003, 2012, and 2015. I also d...
ETS Research Report Series, 2019
Since its 1947 founding, ETS has conducted and disseminated scientific research to support its pr... more Since its 1947 founding, ETS has conducted and disseminated scientific research to support its products and services, and to advance the measurement and education fields. In keeping with these goals, ETS is committed to making its research freely available to the professional community and to the general public. Published accounts of ETS research, including papers in the ETS Research Report series, undergo a formal peer-review process by ETS staff to ensure that they meet established scientific and professional standards. All such ETS-conducted peer reviews are in addition to any reviews that outside organizations may provide as part of their own publication processes. Peer review notwithstanding, the positions expressed in the ETS Research Report series and other published accounts of ETS research are those of the authors and not necessarily those of the Officers and Trustees of Educational Testing Service.
Journal of Intelligence, 2018
Although personality and cognitive ability are separate (sets of) constructs, we argue and demons... more Although personality and cognitive ability are separate (sets of) constructs, we argue and demonstrate in this article that their effects are difficult to tease apart, because personality affects performance on cognitive tests and cognitive ability affects item responses on personality assessments. Cognitive ability is typically measured with tests of items with correct answers; personality is typically measured with rating-scale self-reports. Oftentimes conclusions regarding the personality-ability relationship have as much to do with measurement methods as with construct similarities and differences. In this article, we review key issues that touch on the relationship between cognitive ability and personality. These include the construct-method distinction, sources of test score variance, the maximal vs. typical performance distinction, and the special role for motivation in low-stakes testing. We review a general response model for cognitive and personality tests that recognizes those sources of test score variance. We then review approaches for measuring personality through performance (objective personality tests, grit game, coding speed, economic preferences, confidence), test and survey behavior (survey effort, response time, item position effects), and real-world behavior (study time, registration latency, behavior residue, and social media). We also discuss ability effects on personality tests, indicated by age and cognitive ability effects, anchoring vignette rating errors, and instructions to 'fake good'. We conclude with a discussion of the implications for our understanding of personality and ability differences, and suggestions for integrating the fields.
Journal of Intelligence, 2016
The purpose of this paper is to review some of the key literature on response time as it has play... more The purpose of this paper is to review some of the key literature on response time as it has played a role in cognitive ability measurement, providing a historical perspective as well as covering current research. We discuss the speed-level distinction, dimensions of speed and level in cognitive abilities frameworks, speed-accuracy tradeoff, approaches to addressing speed-accuracy tradeoff, analysis methods, particularly item response theory-based, response time models from cognitive psychology (ex-Gaussian function, and the diffusion model), and other uses of response time in testing besides ability measurement. We discuss several new methods that can be used to provide greater insight into the speed and level aspects of cognitive ability and speed-accuracy tradeoff decisions. These include item-level time limits, the use of feedback (e.g., CUSUMs), explicit scoring rules that combine speed and accuracy information (e.g., count down timing), and cognitive psychology models. We also review some of the key psychometric advances in modeling speed and level, which combine speed and ability measurement, address speed-accuracy tradeoff, allow for distinctions between response times on items responded to correctly and incorrectly, and integrate psychometrics with information-processing modeling. We suggest that the application of these models and tools is likely to advance both the science and measurement of human abilities for theory and applications.
Computers in Human Behavior, 2017
Journal of Intelligence, 2017
Complex problem solving (CPS) has emerged over the past several decades as an important construct... more Complex problem solving (CPS) has emerged over the past several decades as an important construct in education and in the workforce. We examine the relationship between CPS and general fluid ability (Gf) both conceptually and empirically. A review of definitions of the two factors, prototypical tasks, and the information processing analyses of performance on those tasks suggest considerable conceptual overlap. We review three definitions of CPS: a general definition emerging from the human problem solving literature; a more specialized definition from the “German School” emphasizing performance in many-variable microworlds, with high domain-knowledge requirements; and a third definition based on performance in Minimal Complex Systems (MCS), with fewer variables and reduced knowledge requirements. We find a correlation of 0.86 between expert ratings of the importance of CPS and Gf across 691 occupations in the O*NET database. We find evidence that employers value both Gf and CPS skil...
Applied Measurement in Education, 2016
ETS Research Report Series, 2016
Since its 1947 founding, ETS has conducted and disseminated scientific research to support its pr... more Since its 1947 founding, ETS has conducted and disseminated scientific research to support its products and services, and to advance the measurement and education fields. In keeping with these goals, ETS is committed to making its research freely available to the professional community and to the general public. Published accounts of ETS research, including papers in the ETS Research Report series, undergo a formal peer-review process by ETS staff to ensure that they meet established scientific and professional standards. All such ETS-conducted peer reviews are in addition to any reviews that outside organizations may provide as part of their own publication processes. Peer review notwithstanding, the positions expressed in the ETS Research Report series and other published accounts of ETS research are those of the authors and not necessarily those of the Officers and Trustees of Educational Testing Service.
ETS Research Report Series, 2006
The Standardized Letter of Recommendation (SLR), a 28-item form, was created by ETS to supplement... more The Standardized Letter of Recommendation (SLR), a 28-item form, was created by ETS to supplement the qualitative rating of graduate school applicants' nonacademic qualities with a quantitative approach. The purpose of this study was to evaluate the following psychometric properties of the SLR using the Rasch rating-scale model: dimensionality, reliability, item quality, and rating category effectiveness. Principal component and factor analyses were also conducted to examine the dimensionality of the SLR. Results revealed (a) two secondary factors underlay the data, along with a strong higher order factor, (b) item and person separation reliabilities were high, (c) noncognitive items tended to elicit higher endorsements than did cognitive items, and (d) a 5-point Likert scale functioned effectively. The psychometric properties of the SLR support the use of a composite score when reporting SLR scores and the utility of the SLR in higher education and in admissions.
Journal of Experimental Psychology: General, 1991
Universal Energy Systems 4401 Dayton-Xenla Road AJ 4 NOTICE When Government drawings, specificati... more Universal Energy Systems 4401 Dayton-Xenla Road AJ 4 NOTICE When Government drawings, specifications, or other data are used for any purpose other than in connection with a definitely Government-related procurement, the United States Government incurs no responsibility or any obligation whatsoever. The fact that the Government may have formulated or in any way supplied the said drawings, specifications, or other data, Is not to be regarded by Implication, or otherwise In any manner construed, as licensing the holder, or any other person or corporation; or as conveying any rights or permission to manufacture, use, or sell any patented invention that may in any way be related thereto. The Public Affairs Office has reviewed this paper, and it Is releasable to the National Technical Information Service, where it will be available to the general public, including foreign nationals. This paper has been reviewed and is approved for publication.
ETS Research Report Series, 2011
The report presents the findings of a research project funded by and carried out under the auspic... more The report presents the findings of a research project funded by and carried out under the auspices of the Graduate Record Examinations Board.
ETS Research Report Series, 2007
In an effort to standardize academic application procedures, the Standardized Letter of Recommend... more In an effort to standardize academic application procedures, the Standardized Letter of Recommendation (SLR) was developed to capture important cognitive and noncognitive qualities of graduate school candidates. The SLR consists of seven scales (knowledge, analytical skills, communication skills, motivation, self-organization, professionalism and maturity, and teamwork) and was applied to an intern-selection scenario. Both professor ratings (N = 414) during the application process and mentor ratings of the selected students (N = 51) after the internship was completed were collected using the SLR. A multidimensional Rasch investigation suggests that the seven scales of the SLR displayed satisfactory psychometric properties in terms of reliability, model fit, item fit statistics, and discrimination. The two cognitive scales, knowledge and analytical skills, were found to be the best predictors for intern selection. The professor ratings and mentor ratings had moderate to high correlations, with the professor ratings being systematically higher than the mentor ratings. Possible reasons for the rating discrepancies are discussed. Also, implications for how the SLR can be used and improved in other selection situations are suggested.
Uploads
Papers by Patrick Kyllonen