EXPLORING THE FINDINGS
These findings come from a deep exploration of design processes used by people with various levels of expertise. We gathered verbal protocol data from a large number of people who solved design problems while talking out loud. The goal of this work was to understand how individuals with different levels of expertise engaged in design processes. The differences we could see between novice and expert designers would enable us to develop ways to teach about expert design behaviors and how designers can think deeply about the broad implications of their designs.
Here I will briefly describe the findings from a subset of these data, 69 individuals who designed a playground for a fictitious neighborhood. They each individually solved this paper-and-pencil design problem in a lab-based setting in 2 to 3 hours. We evaluated their solution quality and transcribed what they said verbatim. We then segmented that text into phrases, and assigned a design activity code to each phrase to capture what part of the design process they were engaging in. The design activity codes were a way to describe the actions a designer took at a given point in time. The codes we used were generated by looking at engineering design textbooks. The final codes used in this work can be seen on the left in Figure 1 (note that the first and last codes, “identify a need” and “implementation” are grayed out because they were not used in this design task.)
Figure 1. Design activity codes and example timeline.
We collected data from 26 incoming first-year engineering students, 24 graduating engineering students, and 19 engineering design experts. Looking at the student data we found that the graduating seniors had higher quality designs and more sophisticated design processes. As a way to visualize these results, we created timelines from the design data for each participant. A sample timeline is presented on the right in Figure 2. There is one design code for each line in the figure. Time goes from left to right, and a tickmark is put on the appropriate line when a participant says something related to each code. The example shows that a slash was put on the “information gathering” line when this participant said “Hmm, do you have a list of materials.” The timeline representations are helpful displays to identify patterns of time allocation across design activities.
Figure 2. Selected timelines for first-year and graduating senior engineering students.
Figure 2 gives a sense of the overall findings as it presents three first-year and three graduating senior timelines (one each for a low, medium and high quality playground design.) Here we focus on the process of two students: the first-year student with the middle quality playground solution and the graduating student with the high quality design. The first year student with the middle quality design did not spend much time at the beginning doing problem scoping (gathering information, generating ideas) and spent the majority of the time in solid blocks of modeling (or prototyping) without much iteration or transitioning to other design activities.
In contrast, the example graduating senior who created the high quality playground spent a significant chunk of time at the beginning of their process in problem scoping—gathering information and generating ideas as they endeavored to understand the problem they were solving. They continued to gather information throughout the process—a rich set of transitions and iterations throughout their design process demonstrates their agile movement across design activities. This student displayed an overall distribution of activities that goes from the upper left to the bottom right of the timeline—what we are calling a “cascade shape”—as presented in Figure 3. When presented with this data in class, one mechanical engineering student sketched the shape and called it an “ideal project envelope.”
Figure 3. Cascade, or “Ideal Project Envelope” shape.
Figure 4 adds timelines from three experts to the display. Here the data show that there is less variability across the three example timelines. On the whole, the experts spent longer solving the problem than the students did. They also considered more possible solutions to the problem and scoped the problem more effectively by gathering more information and covering more categories of information than the students.
Figure 4. Selected timelines for first-year, graduating seniors and engineering design experts.
Figure 5 uses the high artifact quality graduating senior's design timeline as a canvas to present the overall findings of this research. Here we highlight specific behaviors that demonstrate moving towards more expert design behavior.
Problem scoping before focus on modeling:
Gathering information throughout process
Stay the course (persistence)
Figure 5. Moving towards expert design behaviors.
One of the authors (Cindy Atman) started this research program with funding from the National Science Foundation in 1992. Jennifer Turns joined the team when Cindy moved to the University of Washington in 1998. Over the decades, we have had the privilege to work with amazing collaborators, co-authors, and research team members including Robin Adams, Arif Ahmer, Brad Arneson, Theresa Barker, Maria Buan, Emma Bulojewski, Mary Besterfield-Sacre, Jim Blair, Carie Bodle, Laura Bogusch, Jim Borgford-Parnell, Karen Bursic, Ryan Campbell, Monica Cardella, Soomin Chang, Justin Chimka, Dharma Dailey, Kate Deibel, Zach Goist, Brian Hayes, Melissa Jones, Aaron Joya, Allison Kang, Deborah Kilgore, Kristina Krause, Vipin Kumar, Alex Lew, Terri Lovins, Stefanie Lozito, Janet McDonnell, Kenya Mejia, Annegrete Mølhave, Andrew Morozov, Susan Mosborg, Carie Mullins, Heather Nachtmann, Wai Ho Ng, Will Richey, Eddie Rhone, Axel Roesler, Wendy Roldan, Jason Saleem, Giovanna Scalone, Kathryn Shroyer, Elvia Sierra-Badillo, Shaunte Smith, Roy Sunarso, Steve Tanimoto, Jennifer Turns, Hannah Twigg-Smith, Cheryl Wang, Ken Yasuhara, Mark Zachry, and Eileen Zhang.
This work was supported by National Science Foundation grants 9358516, 9714459, 9872498, 012554, 0227558, and 0354453; the Center for Engineering Learning & Teaching at the University of Washington, the Helmsley Charitable Trust, the Mitchell T. and Lella Blanche Bowie Endowment and the Mark and Carolyn Guidry Foundation.
FOR MORE INFORMATION
This paper provides a comprehensive description of the findings from several decades of research on design expertise.
Scalone, G., Joya, A. J., Shroyer, K. and Atman, C.j. (2019). Design Intentions: Teaching engineering students to plan their design processes. Paper presented at the 2019 American Society for Engineering Education Conference, Tampa, FL, June 5-19 (This paper won the best paper award for the ASEE Design Engineering Education Division (DEED)).
This paper describes how engineering students respond to learning about design expertise through timeline representations.
Atman, C.J., Adams, R. S., Cardella, M. E., Turns, J., Mosborg, S., and Saleem, J. J. (2007). “Engineering Design Processes: A Comparison of Students and Expert Practitioners.” Journal of Engineering Education, 96(4), 359-379.
This is the archival paper for the expert data analysis for the playground design task.
Adams, R. S., Turns, J., and Atman, C.J. (2003). “Educating Effective Engineering Designers: the Role of Reflective Practice.” Design Studies, 24(3), 275-294. (This paper won the Design Studies Best Paper 2003 award.)
This paper presents findings from our studies through Donald Schon’s lens of the reflective practitioner.
Atman, C.J., Chimka, J. R., Bursic, K.M., and Nachtmann, H.L. (1999). “A comparison of Freshman and Senior Engineering Design Processes.” Design Studies, 20(2), 131-152.
This is the archival paper for the student data analysis for the playground design task.