Understanding conceptual electronic design using protocol analysis

Research in EngineeringDesign (1998)I0:129-140 © 1998 Springer-VerlagLondonLimited Researchin Understanding Conceptual Electronic Design Using Protocol Analysis* Thomas Mc Neill 1, John S. Gero2 and James Warren3 1Departmentof ElectricalEngineering,MahidolUniversity,NakornPathom,Thailand;2KeyCentre of Design Computing,Departmentof Architecturaland Design Science,Universityof Sydney,NSW, Australia;3Schoolof Computerand InformationScience,The Universityof South Australia,SA, Australia Abstract. Many theories about the process, of design have been derived from prescriptive or descriptive analysis. There have been few attempts to empirically test these theories. Protocol analysis facilitates detailed inspection of the design process allowing design task analysis across the temporal dimension. Some hypotheses about the design process, derived from the literature, are objectively tested using results obtained from p~vtocol analyses of various electronics designers engaged in conceptual design, Support for the hypotheses is found. K e y w o r d s : Design process; Design protocols; Design research; Function-behaviour-structure model 1. Introduction Much of what has been written about design in the last three decades is characterized by one of three types of analysis: prescriptive, descriptive and investigative. Prescriptive analysis, of which Asimow [1], Mead and Conway [2], Hubka [3], Pahl and Beitz [4], VDI [5] and French [6] axe examples, presents an idealized account of the design process which prescribes the steps that should be followed by a designer. In general, the authors are presenting what they believe to be the best approach to the design of a system o1" artifact. Descriptive approaches [7-9] describe the process based on introspective or anecdotal evidence. Authors of this approach argue that the process which designers actually follow is quite different from the idealized prescriptive account, but the strength of this argument is diminished by the fact that it is based on anecdotal evidence. Investigative analysis [10-14] uses techniCorrespondence and offprint requests to: Professor J. S. Gero, Architectural and DesignScience, Universityof Sydney,Sydney, NSW 5095, Australia. *~his work was carded out at the Key Centre of Design Computing, Universityof Sydney. ques developed in psychology and behavioral science to make objective studies of the design process. One area of data collection that is particularly effective when applied to design is protocol analysis [15]. A designer is asked to think aloud whilst performing a design task and his or her actions and verbalizations are recorded for later analysis. Using protocol analysis, it is possible to observe the design process in detail. In particular, it is possible to observe the temporal aspects of the design process. Common to the three approaches is the desire among researchers to bring about a formal basis for design methodology [16]. In engineering education, the approach taken to teaching design presents the student with a series of design tasks. The student is expected to glean the essential elements of the process by experience. There is little scope for objective evaluation of the process. This is in stark contrast to the learning of other facets of engineering such as electronics, where a range of physical principles based on electromagnetic theory and quantum mechanics forms the basis upon which the subject is learnt and against which it can be evaluated. Dixon [16] advocates the development of scientific theories of engineering design and sees much of the current design research as being in a pre-theory stage. Science develops theories to explain natural phenomena. A scientific theory defines an ideal system which can be used to predict some observable behavior. Dixon makes the distinction between data or observations from the real system, generalizations of the data and explanations of why the generalizations are what they are. A model can be derived from a generalization of the data, but it is theory that explains why the model behaves as it does. Current research in engineering design is at the stage of collecting data, making observations of the design process and finding appropriate generalizations of the data and observations. 130 The work reported here takes some generalizations proposed by authors of the prescriptive and descriptive approaches and derives some hypotheses that can be objectively tested using protocol analysis. This work facilitates the development of a model of the design process which can in the future serve as the basis for the development of theories about engineering design. 1.1. Conceptual Design The earliest phase of the design process is distinguishable from later phases in that it is concerned more with understanding the problem and making general, rather than specific, decisions about the solution [6,17,18]. This phase, referred to as conceptual design, is the topic of study for this paper. Conceptual design commences with a high level description of a requirement and proceeds to a high level description of a solution. The process may include periods of detailed design that are required to elaborate an aspect of a candidate solution to the point where its feasibility can be confirmed. 1.2. Hypotheses Protocol analysis is used to test some hypotheses about conceptual electronic design. Two main hypotheses are presented with some minor hypotheses attached to the second hypothesis. 1. Conceptual design commences with the designer considering mainly function and finishes with the designer considering mainly structure. 2. In the process of designing, the designer cycles between the three activities of analyzing the problem, proposing a solution or partial solution, and evaluating a proposed solution. (a) The designer cycles between the three activities in the order of analysis of the problem followed by synthesis of a solution followed by evaluation of the solution. (b) Conceptual design begins with the designer spending more time on analysis of the problem, followed by a period characterized by mainly synthesis, followed by a period characterized by mainly evaluation of the problem. T. Mc Neitl et aL 2. Protocol Analysis The work described in this paper builds on work undertaken in preliminary studies by the authors. Three protocol studies of engineers engaged in conceptual design were used as a vehicle for the development of the analysis method used here. Two protocol studies of electronic engineers were conducted. Details of the first study appear in Mc Neill and Edmonds [19]. Details of the second stud}' appear in Gero and Mc Neill [20]. A third study was made of an engineer engaged in mechanical design [21]. When analyzing the protocol data, two important factors are to be taken into consideration. The coding scheme used must adequately reflect the complexity of the data without distorting it, and the process of coding the protocol must be as objective as is possible. The first of these considerations can be addressed by a combination of coding categories that are derived from real data collected in the earlier studies [20], and categories that are derived from theories of design. The second consideration can be addressed by the use of two coders and a consensus approach to the final coding result. There have been criticisms of the use of the think aloud protocol analysis as a means of producing a unitary cognitive model of human designers [22]. The current approaches offer the opportunity to gain some insight into a range of design processes of humans without claiming complete insight. As such, they are useful in the development of cognitively-based models of design. 2.1. Coding Scheme The coding scheme makes use of the notion of multiple dimensions by using up to four codes to capture the aspects of each coding segment. Two dimensions are concerned with the designer's navigation within the problem domain, and two dimensions are concerned with the strategies used by the designer. The designer's interaction with the design problem can be characterized by navigation through a two dimensional space representing the problem domain. One dimension is the level of abstraction, and the other is concerned with whether the designer is reasoning about function, behavior or structure. The designer can be considering the problem at a level of abstraction that ranges from a high level to a detailed level. Between these two extremes an arbitrary number of levels may be identified. Watts [23] represents the range as a continuum between concrete and abstract. Lang and Mc Cormick [24] and