Bruno Zumbo

Followers

Following

Public Views

Interests

Uploads

Papers by Bruno Zumbo

A dialectic on validity: Explanation-focused and the many ways of being human

International Journal of Assessment Tools in Education, Dec 26, 2023

In line with the journal volume's theme, this essay considers lessons from the past and visions f... more In line with the journal volume's theme, this essay considers lessons from the past and visions for the future of test validity. In the first part of the essay, a description of historical trends in test validity since the early 1900s leads to the natural question of whether the discipline has progressed in its definition and description of test validity. There is no single agreed-upon definition of test validity; however, there is a marked coalescing of explanation-centered views at the meta-level. The second part of the essay focuses on the author's development of an explanation-focused view of validity theory with aligned validation methods. The confluence of ideas that motivated and influenced the development of a coherent view of test validity as the explanation for the test score variation and validation is the process of developing and testing the explanation guided by abductive methods and inference to the best explanation. This description also includes a new re-interpretation of true scores in classical test theory afforded by the author's measure-theoretic mental test theory development-for a particular test-taker, the variation in observed test-taker scores includes measurement error and variation attributable to the different ecological testing settings, which aligns with the explanation-focused view wherein item and test performance are the object of explanatory analyses. The final main section of the essay describes several methodological innovations in explanation-focused validity that are in response to the tensions and changes in assessment in the last 25 years.

Download

Getting the most out of intensive longitudinal data: a methodological review of workload–injury studies

BMJ Open, Oct 1, 2018

Objectives To systematically identify and qualitatively review the statistical approaches used in... more Objectives To systematically identify and qualitatively review the statistical approaches used in prospective cohort studies of team sports that reported intensive longitudinal data (ILD) (>20 observations per athlete) and examined the relationship between athletic workloads and injuries. Since longitudinal research can be improved by aligning the (1) theoretical model, (2) temporal design and (3) statistical approach, we reviewed the statistical approaches used in these studies to evaluate how closely they aligned these three components. Design Methodological review. Methods After finding 6 systematic reviews and 1 consensus statement in our systematic search, we extracted 34 original prospective cohort studies of team sports that reported ILD (>20 observations per athlete) and examined the relationship between athletic workloads and injuries. Using Professor Linda Collins' three-part framework of aligning the theoretical model, temporal design and statistical approach, we qualitatively assessed how well the statistical approaches aligned with the intensive longitudinal nature of the data, and with the underlying theoretical model. Finally, we discussed the implications of each statistical approach and provide recommendations for future research. results Statistical methods such as correlations, t-tests and simple linear/logistic regression were commonly used. However, these methods did not adequately address the (1) themes of theoretical models underlying workloads and injury, nor the (2) temporal design challenges (ILD). Although time-to-event analyses (eg, Cox proportional hazards and frailty models) and multilevel modelling are better-suited for ILD, these were used in fewer than a 10% of the studies (n=3). Conclusions Rapidly accelerating availability of ILD is the norm in many fields of healthcare delivery and thus health research. These data present an opportunity to better address research questions, especially when appropriate statistical analyses are chosen.

Download

Testing for Sample Invariance of IRM Item Parameter Estimates: A case study on real response data

Download

Reliability, Power, Functions, and Relations: A Reply to Humphreys

Applied Psychological Measurement, Mar 1, 1993

Download

Assembled validity: rethinking Kane’s argument-based approach in the context of International Large-Scale Assessments (ILSAs)

Assessment in Education: Principles, Policy & Practice, Nov 1, 2020

Drawing on Kane's argument-approach to validity and Toulmin's later work on cosmopolitanism and d... more Drawing on Kane's argument-approach to validity and Toulmin's later work on cosmopolitanism and diversity, this paper asks whose validity arguments and evidence are being presented in International Large-Scale Assessments (ILSAs), where and when. With a case study of the OECD's PISA for Development, we demonstrate that validity arguments are assembled, negotiated and transformed by the network of actors who come together in ILSAs. We claim that the challenge of ILSAs is not to establish a single authoritative argument through the displacement of plural interpretations and uses. We argue that one of the tasks of an argument-based approach to validity is to create a democratic space in which legitimately diverse arguments and intentions can be recognised, considered, assembled and displayed. We therefore suggest that 1) this socio-material practice of assembling validity should be integrated into validity theory and practice, and 2) the task of assembling validity should be informed by democratic principles of diversity and inclusion.

Download

Designer's Corner - Translating and Adapting Measurement Instruments for Cross-Linguistic and Cross-Cultural Research: A Guide for Practitioners

Canadian Journal of Nursing Research Archive, Jun 1, 2003

The psychometric instruments used in cross-linguistic or cross-cultural research are typically de... more The psychometric instruments used in cross-linguistic or cross-cultural research are typically developed in one language and then translated into another.The authors address methodological problems that arise in the translation process and that compromise data interpretation, using concrete examples to illustrate these problems.They point out and describe the relevance of lexical semantics in item translation.The authors make recommendations for avoiding common pitfalls in the use of translated measurement instruments or in the translation or adaptation of such instruments. This paper is intended for researchers who are planning to develop or use translated instruments. Measurement is integral to nursing practice. It is used to facilitate the diagnosis of physical and psychological health problems and in the assessment of pain as well as physical and cognitive functioning. It can also serve a useful purpose in the assessment of self-reported health status, knowledge or attitudes about illness, and health services or policies. Measurement instruments may take a variety of forms, including questionnaires, tests, rating scales, and self-reports.The choice depends largely on the purpose of the research that is undertaken. Psychometric measures typically hinge upon a focal theme or concept. For instance, a performance test may be designed to assess a cognitive ability such as attention or memory. An attitudinal scale may quantify individuals' disposition towards smoking or their impression of services dispensed by a healthcare institution.A questionnaire may serve to assess older persons' quality of life. All themes or concepts generally involve various facets, which are measured using different questions or items.The potential range of applications of such measures is open-ended.

Download

The Role of Item Distributions on Reliability Estimation: The Case of Cronbach’s Coefficient Alpha

Educational and Psychological Measurement, Feb 11, 2020

Simulations concerning the distributional assumptions of coefficient alpha are contradictory. To ... more Simulations concerning the distributional assumptions of coefficient alpha are contradictory. To provide a more principled theoretical framework, this article relies on the Fre ´chet-Hoeffding bounds, in order to showcase that the distribution of the items play a role on the estimation of correlations and covariances. More specifically, these bounds restrict the theoretical correlation range [21, 1] such that certain correlation structures may be unfeasible. The direct implication of this result is that coefficient alpha is bounded above depending on the shape of the distributions. A general form of the Fre ´chet-Hoeffding bounds is derived for discrete random variables. R code and a user-friendly shiny web application are also provided so that researchers can calculate the bounds on their data.

Download

Multi-Group Confirmatory Factor Analysis for Testing Measurement Invariance in Mixed Item Format Data

Journal of Modern Applied Statistical Methods, Nov 1, 2008

This simulation study investigated the empirical Type I error rates of using the maximum likeliho... more This simulation study investigated the empirical Type I error rates of using the maximum likelihood estimation method and Pearson covariance matrix for multi-group confirmatory factor analysis (MGCFA) of full and strong measurement invariance hypotheses with mixed item format data that are ordinal in nature. The results indicate that mixed item formats and sample size combinations do not result in inflated empirical Type I error rates for rejecting the true measurement invariance hypotheses. Therefore, although the common methods are in a sense sub-optimal, they don't lead to researchers claiming that measures are functioning differently across groups -i.e., a lack of measurement invariance.

Download

Heteroskedasticity in Multiple Regression Analysis: What it is, How to Detect it and How to Solve it with Applications in R and SPSS

Practical Assessment, Research and Evaluation, 2019

Within psychology and the social sciences, Ordinary Least Squares (OLS) regression is one of the ... more Within psychology and the social sciences, Ordinary Least Squares (OLS) regression is one of the most popular techniques for data analysis. In order to ensure the inferences from the use of this method are appropriate, several assumptions must be satisfied, including the one of constant error variance (i.e. homoskedasticity). Most of the training received by social scientists with respect to homoskedasticity is limited to graphical displays for detection and data transformations as solution, giving little recourse if none of these two approaches work. Borrowing from the econometrics literature, this tutorial aims to present a clear description of what heteroskedasticity is, how to measure it through statistical tests designed for it and how to address it through the use of heteroskedastic-consistent standard errors and the wild bootstrap. A step-by-step solution to obtain these errors in SPSS is presented without the need to load additional macros or syntax. Emphasis is placed on the fact that non-constant error variance is a population-defined, model-dependent feature and different types of heteroskedasticity can arise depending on what one is willing to assume about the data.

Download

GrOup based physical Activity for oLder adults (GOAL) randomized controlled trial: study protocol

BMC Public Health, Jun 27, 2015

Background: Physical activity has health benefits across the lifespan, yet only 13 % of Canadian ... more Background: Physical activity has health benefits across the lifespan, yet only 13 % of Canadian older adults are sufficiently active. Results from a number of observational studies indicate that adults display positive preferences for exercising with others of a similar age and same gender, and that intra-group age-and gender-similarity are associated with elevated exercise adherence. However, research has yet to experimentally examine the extent to which intra-group age-and gender-related similarity affect exercise adherence behaviors. Methods/design: The GrOup-based physical Activity for oLder adults (GOAL) trial is a three-arm randomized control trial that will examine the efficacy of two different group-based exercise programs for older adults (informed by the tenets of self-categorization theory) in relation to a standard group-based exercise program. Within this manuscript we outline the design and proposed evaluation of the GOAL trial. The first arm is comprised of exercise groups made up of participants of a similar-age and of the same gender; the second arm consists of groups with similar-aged mixed gender participants; the control arm is comprised of mixed-aged mixed gender participants. We aim to compare the adherence rates of participants across conditions, as well as potential moderation effects and mediating mechanisms. Discussion: Results from this trial will inform intervention designs to improve the exercise adherence behaviors of older adult. At a systems-level, should support be derived for the efficacy of the interventions tested in this trial, changing group composition (i.e., age, gender) represents a feasible program adaptation for physical activity centers.

Download

A New Nonparametric Levene Test for Equal Variances

Psicologica, Jun 1, 2010

Tests of the equality of variances are sometimes used on their own to compare variability across ... more Tests of the equality of variances are sometimes used on their own to compare variability across groups of experimental or non-experimental conditions but they are most often used alongside other methods to support assumptions made about variances. A new nonparametric test of equality of variances is described and compared to current 'gold standard' method, the median-based Levene test, in a computer simulation study. The simulation results show that when sampling from either symmetric or skewed population distributions both the median based and nonparametric Levene tests maintain their nominal Type I error rate; however, when one is sampling from skewed population distributions the nonparametric test has more statistical power.

Download

Extensions Thereof

… an item shows DIF if individuals having the same ability, but from different groups, do not hav... more … an item shows DIF if individuals having the same ability, but from different groups, do not have the same probability of getting the item right (Hambleton, Swaminathan, & Rogers, 1991, p. 110) Let us begin in the beginning …. with binary item responses. ! Uniform DIF exists when the probability of answering the item correctly is greater for one group than the other uniformly over all levels of ability: There is no interaction between ability level and group membership. ! Nonuniform DIF exists when the probability of answering the item correctly is not greater across all levels of ability for any group: There is interaction between ability level and group membership.

Download

The Role of Item Distributions on Reliability Estimation: The Case of Cronbach’s Coefficient Alpha

Educational and Psychological Measurement, 2020

Simulations concerning the distributional assumptions of coefficient alpha are contradictory. To ... more Simulations concerning the distributional assumptions of coefficient alpha are contradictory. To provide a more principled theoretical framework, this article relies on the Fréchet–Hoeffding bounds, in order to showcase that the distribution of the items play a role on the estimation of correlations and covariances. More specifically, these bounds restrict the theoretical correlation range [−1, 1] such that certain correlation structures may be unfeasible. The direct implication of this result is that coefficient alpha is bounded above depending on the shape of the distributions. A general form of the Fréchet–Hoeffding bounds is derived for discrete random variables. R code and a user-friendly shiny web application are also provided so that researchers can calculate the bounds on their data.