Quantity qualifies quality (.pdf)

Quantity qualifies quality Ali Baykala,1* a Bogazici University, 34342 Istanbul, Turkey Abstract A simple analysis of papers submitted in current educational research journals shows that educational researchers shy away from representing the results of their research in quantitative forms. Qualitative approaches have long dominated the field of education. This is because the usual quantitative methods fail to reflect the important qualitative aspects of effective and efficient educational practice. The main reason is the error of isomorphism prevail the quantitative operations applied in data analyses. Statistical or mathematical research methods in education must also comply with the same genuine scientific measurement principles as in the other disciplines. This doesn’t mean that researchers may be insensitive to the essential qualitative aspects of high- quality educational research. Shortly quantity does not alternate or replace quality. Quality is not the antonym of but just the conjugate of quality. Keywords: Data quality, qualitative evaluation, quantitative evaluation, implicit quantifiers 1. INTRODUCTION 1.1 Why do we need data? Educational decisions to be made based on evidence are vitally important for the individuals and for the public. Understanding and disseminating evidence is more important than they have ever been. Nevertheless analysis of educational data is intrinsically intricate. It involves myriad of variables. All major variables are interrelated among themselves and intervene with others. Linearity of relations is mostly an assumption rather than the reality. Speed and accuracy are essential for data collection, simplicity and continuity are indispensable for its presentation to the interested parties. High levels of literacy and numeracy are expected of data consumers. The need for educators and for their audience to reason with complex data goes beyond just intellectual value of numerical skills. Competencies to extract hidden practical implications and to construct theoretical meaning need to be developed that can be applied in a range of instructional contexts. In order to design, implement and evaluate the curriculum in an area and coordinate curricula across subjects must be based on evidence. In understanding evidence conceptual analysis is essential. Otherwise professional judgment tends to be obscured by inexpert experimentalism. Making complexities and uncertainties in educational processes intelligible to the stake holders requires high quality data. For example drawing inferences and making predictions about the student progress cannot be accomplished without having sophisticated data representations. Simple data sets are not sufficient for educators who must reason with multivariate data (Ridgway, Nicolson, McCusker; 2006). Since facts replace guesses and/or wishful thinking data adds legitimacy to discussions and explanations of educational problems therefore every educational assertion must be backed by data or research, 1.2 Data explosion The pressure on individuals and on institutions to achieve high performance as measured by central, high-stakes examinations is increasing day by day. Thus the assessment area is dynamic with an explosion of new theories and practices. Achievement motivation and * Corresponding author. Tel.: +90-212-3594558; fax: +90-212-2575036. E-mail address: [email protected]. avoidance of failure in nation wide examinations enforce educators to use data to make informed decisions about instructional practice that may lead to improving achievement. Increase in testing results with an explosion of data which need to be the more pressures on practitioners. The more data the louder excuse by teachers to spare less time for instructional design and its implementation. Teachers complain that when they are forced to teach to the tests they have to ignore some topics and waste their energy to be utilized in instructional design and its implementation. Shortly the more time on test means the less time on task. In fact teachers are familiar with using data for instructional decisions and planning. Even those who have not received pre-service training on how to use data in the classroom are influenced by the accent that their peers put on using data. They also recognize the significance of using multiple sources of data rather than relying on a single source. This is almost spontaneous when teachers examine central exam scores. Teachers in an assessment system must be managed properly to motivate for producing maximum yield from the data (Mandinach, 2006). The word “system” in the term “assessment system” is not used arbitrarily. On the contrary, it is a technical concept that is both descriptive and explanatory. General systems theory rests on the assumption that parts comprising the whole affect each other. It allows for the observation of interactions between different components of the whole instead of segregation of isolated elements so that the whole is more or less than the sum of its parts. (Katz and Kahn, 1966). This implies that, one should go beyond the individual characteristics of components to comprehend the complexity of the entire system. Systems thinking is not an alternative methodology by which one can design better, more effective or more efficient assessment practices. It is a paradigm which enables the designer to describe, explain, predict and prescribe every kind of educational project which may be good or bad for some reason. It is a value-free model by which one can analyze a descriptive or an experimental research. Systems thinking enables the educator to design the structure and dynamics of system as well as analyzing them. There are so many attributes expected of assessment or research designs implicitly or explicitly. Most of these attributes are case specific, biased and subjective. Also these traits compete with each other most of the time. One may give emphasis on the humanitarian aspect of the evaluation process rather than the technological innovation involved. One neither has to comply with any of those myriad of test theories to develop a science exam, nor s/he has to deny any one of them. The least effective and the most efficient assessment practices can be explored with the help of systems concepts, because all evaluation systems, whether we work towards accessing or retreating from them, have structural elements and functional processes. Table 1 depicts a summary of structural elements of an assessment system at school level Table 1. Components of instructional systems and their functions Components Unique function A few examples Input Prerequisite for all, Problems, theories, funds, determinant of all Physical Accommodate all the other Data warehouse, testing rooms, Settings components observatories Social Consent the existence of the Student, teacher, proctor, Settings instructional system administrator, parent Media and Store and retrieve stimuli Paper-pencil tests, computers, materials relevant to desired output testing kits Methods and Sequence interactions relevant Tests, interviews, exams, processes to desired output questionnaires Coordinator Coordinates and controls all Teacher, proctor, trainer, the others mentor, observer Outputs Frame of reference for all quality data, research findings 1.3 Data is necessary but not sufficient The ultimate aim of education is to achieve wisdom. Wisdom is the focus of converging knowledge and diverging reflections. The evolution of wisdom starts from raw data. The sequence of "data-information-knowledge-wisdom" along the experiential continuum displays not a linear but spiraling growth. There will be iterations between the stages depending upon the content of interaction. The goal is to translate data to information and transform information to knowledge. Accumulation of knowledge lays the foundations of wisdom from which reflective thinking, diverging ideas will emerge. This cognitive data processing framework engenders data-driven decision making. The significance of this framework is the spiraling nature of iterations that transform data into information and ultimately to actionable knowledge. The dynamic interactions of interconnected structural components in real instructional practice can be depicted by systems thinking. 1.4 How to use data Periodical assessments must be developed to measure school and student progress so that educators use data to help all students improve their learning. Informed instructional decisions are based on complex and diverse sources of data. Improvement of instruction and student learning require technology-based solutions. Technology-based tools facilitate decision making. Administrators and teachers who can take the advantage of these tools enhance instruction. They can also appreciate how assessment data can be used effectively in all phases of educational process. Maintaining systems view as the primary conceptual perspective in data driven applications they can bring qualitative and quantitative techniques together. There are technical limitations in assembling and disseminating data across complex systems therefore the research literature on data to support instructional decision making at the classroom level is quite limited. Difficulties in data management (i.e. entry, storage, retrieval, analysis, and presentation) can be overcome by technical facilities. But the quality and interpretation of data, and taking the most of data for instructional practice is the main challenge. Naturally most of the administrators and teachers are experts in other fields than educational testing and measurement. However in every field of specialization a high level of data literacy is needed and some level of assessment literacy will be expected of educators in making data-driven decisions. Firstly they must be willing to use and understand the data. Secondly they must acquire tool skills to utilize the tools to handle data. School administrators are expected to display leadership in producing and maintaining the data culture in the school, teachers are expected to produce high quality data. 2. SOME BIPOLAR ATTRIBUTES OF DATA QUALITY: 2.1 Access vs. security Data must be easily accessible to data cosumers. An up-to-date medium is the Internet. There will be precautions to maintain the data security. There will be different access levels for administrators, teachers, parents and students. When a participant logs in data-storage system must produce a record for each entry (Mandinach, 2006). 2.2 Permanence vs. Transience The length of the period of storage and retrieval of data varies for different types of data. Student attendance must be recorded everyday. The period for quizzes may be left arbitrary to the teachers. Registration records may be updated annually or semester wise. Course grades will be kept forever. Mail addresses can be changed on demand. School performance on central entrance examinations will be appended each year. Some data can be used at the beginning of the year to see the students’ level of academic performance. Assessment scores of classes can also be analyzed at the end of the term to see how each class compares to other class in the school. 2.3 Comprehensibility-Complexity Data Comprehensibility deals with the understandability of the information. how understandable the functioning of the tool is; how clear the presentation of the data are; and how easy it is to make reasonable inferences from the information presented. The more understandable, the more likely the tool will be used. Some parts may be open to misinterpretation and ambiguity even by trained specialists. The multiple forms of graphic representations are intended to make the data readily understandable to different users. Too much emphasis on understandability ends up with oversimplicity. 2.4 Data for validity vs.validity of data In any assessment program a precise definition of what is to be measured and what method of measuring it is the most appropriate are the most crucial issues (Bond, 2004; Guilford, 1965; Hambletone, 1978). Irrelevant assumptions, inconsistent definitions, incorrect evidence can easily be taken for granted all along the assessment program. Item format chosen, values accepted, omitted options, and logical comparisons to prior practice must be reviewed by the expert eye (Rust&Golombok, 2009). Data quality includes but is not limited to validity. For instance classical test theory is internally inconsistent in the quantitative reduction of test data. As the intersubject variability decreases, the reliability coefficient may take minus values which contradicts the conceptual and logical definitions of reliability. Also, reliability is indefinite when the variance diminishes. The desirable magnitude of the item difficulty index suggested by classical theory to optimize the reliability, contradicts the conceptual framework of validity. The performance of a student is defined as the composite of the truth and the error. The shortcomings of the theory stem from the assumption that the error varies randomly within the observed scores. Randomness could be taken as a measure of the error, instead of assuming that the error is random (Shannon and Weaver; 1949; McGill, 1954). Chance success can be eliminated without any extra irrelevant procedure for corrections. New quantifiers can be developed to describe the quality characteristics of a test items, and also inter-item and item-test relationships. No confirmatory factor analysis can correct the semantic inadequacies or distractors embedded in a statement of test item. Can there be linear structural model to correlate the intensity of stimuli with the amount of retention of response reinforced intermittently? The intensity of applause in a ceremonial talk as measured in decibels can hardly be used to interpret the meaning of a joke in its cultural context. Data for the construct validation of a test is mostly a qualitative adventure by its very nature. But the agreement among expert opinion can easily be expressed with a simple proportion. 2.5 Timeliness-Spontaneity Data must be current and timely to be used for decision making. The delay between data collection and the decision making restrains the benefits of evaluations. Especially the time interval between the input and the feedback data is very crucial for educational assessments. On the other hand there are so many not premeditated learning experiences in an educational setting. Therefore the assessment system must be capable of managing spontaneous data which may arise from unplanned instructional endeavors. In classical testing programs steps are sequential and simultaneous for everyone. All of the students start responding to questions at the same time and drop their pencils when the bell rings. Such a timely, preplanned testing is not necessarily the best practice for today. On-line, web-based testing offers subjects self planned assessment schedule. 2.6 Measurement Equivalence vs. Discriminating Measures: It is difficult to compare observed mean scores in raw data without having measurement equivalence. A test or a subtest is said to have measurement equivalence across populations if subjects with identical scores on the construct have the same raw score at the item level, or at the subtest level. Since subjects differ with respect to the construct measured by items discriminating functioning of the item is widespread in test data. In free-format items, each item could evoke a myriad of different responses. However, in analyzing such data, it may be more practical to categorize these responses into a limited number of categories that can be rank ordered in order of attainment or intensity (Raju, Laffitte, Byrne, 2002). 2.7 Quantitative vs qualitative In the past the quantitative research has been recognized as indispensible support for educational experiments and explorations. Recently however, a greater portion of educational research is qualitative in nature (Callingham and Bond, 2006). Why do qualitative approaches appear to dominate this field? Is it because that the usual quantitative methods are unable to discover the important qualitative aspects educational events? Are there some qualities for which the principles of the metric system can be ignored? Can any quantitative research methodology in education not sustain sensitivity to those significant qualitative aspects of educational research? How can the general, applicable, communicable findings of quantitative research be left aside? Some researchers maintain the view that any aspect of the human condition should not be described along a single dimensional scale (Medley, 2000). But the weight of a newborn baby or the height of a basketball player yield some information which cannot be overlooked. They may be insufficient all by themselves for the complete description of physical qualities. Is there a perfect qualitative description of the physical well being? Multiattribute measurement is essential but measurement of each attribute may be useful for some decision. In some cases a single index can be developed to summarize a set of attributes as single quantifier. Human Development Index or ICTs Index are commonplace examples among the many others. Obesity index for instance combines weight and height together and uses that score as a summary of physical and physiological well being. There may be some practical concerns in such classifications. But the purposes for research projects are the same in essence. All scientific studies tend to produce a better descriptions, explanations and predictions about the phenomena within their scope. In qualitative research data are collected in free format styles such as interviews, essays, and observations. There may be no prior methodological reservation for the next coming stages. The researcher discovers patterns and assigns codes to these patterns during the data collection (Callingham and Bond, 2006). These codes are nothing but nominal scale values some of which may comprise a new variable as data accumulates in progress. Formally this is not an endless process. There will be a point in time at which no new patterns appear. Then the researcher stops and reduces data into information. No matter how verbal or pictorial the codes are there will be implicit quantifiers in descriptions, comparisons etc. 3. UNIFYING THE QUALITATIVE QUANTITATIVE ATTRIBUTES Data qualities summarized above ought to be seen from viewpoint of mutual relations and dependencies. The optimal level of these bipolar characteristics reflects the value and soundness of the assessment system. One can find some other unified bipolar features of a data system. A myriad of assessment procedures must be analyzed to discover the bipolarities relevant to a particular research which researchers run into. Bipolarities are discussed with reference to conventional assessment experiences. Some commonplace examples are given for clarification. The focus was on the research concept. The more bipolarities considered, the more complex and unclear the discussion will be. But this is the way to develop our conceptual skills. We must reflect upon at least two sides to every attribute one may come across. Such a reflection requires both theoretical and logical analysis. Theoretical analysis delineates natural world of beings while the logical reasoning deliberates the means and the ends of practical conduct. This is another bipolarity which requires reunion between two forms of reason. One should stay away from self-centered preferences to discriminate between theoretical and logical analysis. Because the overexpansion of either into the realm of the other. This is something which is consistent with the interconnectedness of structural elements. No single attribute can be subordinated to another. We cannot emphasize any quality at the expense of another. Bipolarity suggests that any attribute is composed of two challenging qualities in agreement. A compliant relationship between the two poles as such exhibit a variety within a unity. But bipolarity in open systems is not as simple as unity of opposites. The contradiction does not take place between the competing qualities, but between their extremities. In fact the extremities of desirable attributes of open systems do not qualify but rather turn down the system. In other words whenever a system assumes an extreme polarity of some attribute ceases to exist as a coherent system. Such a logical process has been referred as dialectics. Dialectics dates back to Plato. Hegel elaborated the dialectical thought. In lay conversation dialectics is immediately coupled with Marx and Marxist materialism. In the discussion here there are some differences between the optimization for compliance between the bipolar system attributes, and conventional Hegelian or Marxist dialectical approach. First of all bipolar attributes do not alternate but complement each other. Bipolar attributes are not antagonistic contradictions such as war or peace, life or death, yes or no etc. Only the extremities of bipolar attributes are antagonistic. The educational researcher’s role is to discover the optimal values of bipolar characteristics for a given level of effectiveness and efficiency. Educational research designer is supposed to mediate the bipolar options to bring about a desirable composition. Second difference between the classical dialectical synthesis and the search for compliance is that in the former thesis and antithesis are not separable, but in latter the attributes are at least virtually separable. For instance there are assessments based on measurement excellence, and yet there are some others which purify discrimination. The proposed approach here does not disqualify any quality, but rather attempts to unify them. In other words the bipolar attributes of systems can be identified and defined as independent parameters, but their coordinated standings is more explanatory and prescriptive as well. The fission of bipolar attributes is possible but their fusion is more desirable. 4. CONCLUSION At the end quality level of an assessment can be assessed by the level of reasoning it provides. These levels are parallel to Stevens scales of measurement. Mathematical Level: Complete absence of dimension or construct being measured can be defined. Equidistant or uniform units of measurement exists. Thus proportional or differential comparisons are meaningful. Systematic and constant errors are eliminated, and random error can be tolerated by repetitive measurements. Numerical data is isomorph with the numbers. Mathematical formulas can be developed. All parametric statistics can be performed. Analytical Level: Absolute absence of construct concerned is uncertain. Thus proportional reasoning is missing. There exists an arbitrary zero point which pertains all observations therefore constant error is not systematic and also does not mix with random error. Appropriate use of uniform units is possible. Comparisons can be made in terms of arithmetic differences. Parametric statistics can be performed if their distribution assumptions hold. Exploratory Level: Observations are expressed in terms of ranks or ratings. Comparisons depend on ordinal positions of the subjects based on objective criteria. Non-parametric statistical inferences can be drawn. Assertions involve implicit quantifiers. Classificatory Level: Objective discrimination with respect to a predetermined criterion; subjective selective engagement with context, often in supportive formats, appropriate recognition of conclusions but without verification, frequency distributions, intuitive taxonomies, colloquial or informal engagement with context. Ipsative Level: Idiosyncratic engagement with context, tautological use of terminology, and demonstrating basic mathematical skills rhetorically, rough probabilistic estimations. This hierarchical categorization is just an example of classificatory level reasoning. It is just another way of saying that the quality of reasoning ascends parallel to the increase in the amount of quantity involved. One may quite rightly challenge its designation and exemplary content, because it is a hermeneutic approach to quality versus quantity dilemma. 5. REFERENCES Bond, T. G. (2004). Validity and assessment: a Rasch easurement perspective.Metodologia de las Ciencias del Comportamiento, 5(2), 179–194. Callingham, R. and Bond, T. (2006). “Research in Mathematics Education and Rasch Measurement”. Mathematics Education Research Journal 2006, Vol. 18, No. 2, 1-10 Garman, E. (1999). “Cool data: quantity AND quality”. Biological Crystallography. Acta Cryst. (1999). D55, 1641±1653. Guilford, J.P. (1965). Fundamental Statistics in Psychology and Education. New York: McGraw-Hill, Kogagusha, 1965. Hambleton, R.K. et. al. (1978). "Criterion-Referenced Testing and Measurement." Review of Educational Research. 48:1, (Winter 1978), 1-47. Katz, D., Kahn, R.L.(1966). Common characteristics of open systems. In Emery, F.E. (1970). (Ed.). Systems thinking (pp. 86-104). Middlesex: Penguin Books. Mandinach, E.B., et. al. (2006). The Impact of Data-Driven Decision Making tools on Educational Practice: A Systems Analysis of Six School Districts. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, April 9, 2006. McGill, W.J. (1954). "Multivariate Information Transmission". Psychometrika. 19:2, (June 1954), 97-116. Medley, M.D. (2000). “Using Qualitative Research Software for CS Education Research”. Department of Mathematics and Computer Science Augusta State University. Raju, N.S., Laffitte, L.J., Byrne, B.M. (2002). “Measurement Equivalence: A Comparison of Methods Based on Confirmatory Factor Analysis and Item Response Theory”. Journal of Applied Psychology, Vol. 87, No. 3, 517–529. Ridgway, J., Nicholson, J. McCusker, S. (2006). Reasoning with evidence–new opportunities in assessment. http://www.dur.ac.uk/resources/smart.centre/Publications/ridgwayicots7assess.pdf. (Accessed on April 1st, 2011). Rust, J., Golombok, S. (2009). Modern Psycometrics, (Third edition). London: Routledge, Taylor and Francis. Shannon, C.E., Weaver, W. The Mathematical Theory of Communication. Urbana: The Üniversity of Illinois Press, 1949. 6. ACKNOWLEDGEMENT This study is a part of a research project coded as 6038 and titled as Using Entropy Indices In The Analysis Of Test Data. Project is being supported by Bogazici University Scientific Research fund.