Session T3A ESTABLISHING LEARNING OBJECTIVES AND ASSESSING OUTCOMES IN ENGINEERING SERVICE COURSES?
P. David Fisher1 , James S. Fairweather2 and Lisa A. Haston3
Abstract ? This paper examines the challenges and possible strategies for implementing continuous-qualityimprovement processes in engineering service courses. We describe the process we have been using to review and revise course learning objectives through interactions with various constituent groups. We also describe our outcomes assessment process and the tools developed to outcomes. Key challenges in achieving systemic reform in engineering service courses are faculty buy-in to the reform process and the diversity of outcomes expected by the various constituent groups. Index Terms ? Assessing outcomes, engineering service courses, establishing learning objective, systemic reform. Since mid-1997, an interdisciplinary team of faculty and graduate students has been seeking ways to bring about sustainable educational reform within engineering service courses through a grant from the GE Fund. Progress toward achieving this objective was described in a FIE’99 paper, “Transforming Engineering Service Courses” . One important outcome from this activity was recognition that engineering service courses could provide an important link to the student’s major engineering design experience. This outcome was described in an ASEE’00 paper, “Linking Engineering Service Courses with Engineering Design”  and was the focus of an ASEE’00 workshop, “Linking Service Courses with Design” . This current paper focuses on the course and curricular reform efforts that have taken place during the past three years in one of Michigan State University’s six engineering service courses: ECE 345—Electronic Instrumentation Systems  . This course is intended to introduce nonelectrical and non-computer engineering majors to core topics related to the application of electric circuits, electronics and instrumentation to the monitoring and control of physical processes encountered by these future engineers. In addition to providing important content, engineering service courses and other experiences that take place early in a student’s academic program can foster innovations in subsequent upper division courses, including capstone design. As an example, the highly institutionalized and successful ENES 100 freshman engineering program at the University of Maryland, which integrates communication skills, teamwork, and other collaborative learning approaches to expose students to design, has spawned a new design-based manufacturing course and a new senior capstone design experience . The ENES 100 experience now is available in modified form to non-engineering students through the GEMSTONE program . In both cases, expectations by students and faculty from the ENES experience led to changes in content and pedagogy in later courses.
An engineering service course may be defined as a required or elective course taken by engineering students outside their principal field of study—e.g., an environmental engineering or computer engineering course taken by students majoring in mechanical engineering. Early in 1997, a group of faculty in the College of Engineering at Michigan State University MSU) formed a task force to review the roles of courses in engineering degree programs. The task force came to recognize that engineering service courses were often overlooked—or even discounted—in terms of their potential educational value. This conclusion became evident when the faculty began the process of documenting how educational program objectives were achieved within specific undergraduate engineering programs. By and large, members of MSU’s engineering faculty viewed engineering service courses primarily as a longstanding engineering curricular mandate, promulgated by ABET with the following requirement: “In order to promote breadth, the curriculum must include at least one engineering course outside the major disciplinary area” . The task force began to look beyond this cryptic requirement to add breadth to engineering programs and asked the question: How might engineering services courses at MSU be transformed so that they genuinely impact the educational program outcomes mandated in EC2000’s Criterion 3 [ 2]? This early process was described in an ASEE’98 paper, “Assessment Process at a Large Institution” .
This work was supported in part by the General Electric Fund through a grant entitled “Reforming the Early Undergraduate Engineering Learning Experience.” 1 P. David Fisher, Michigan State University, Dept. of Elect. and Comp. Engr., 2120 Engineering Bldg., East Lansing, MI 48824-1226 firstname.lastname@example.org 2 James S. Fairweather, Michigan State University, Educational Administration, Erickson Hall, East Lansing, MI 48824 email@example.com 3 Lisa A. Haston, Michigan State University, Educational Administration, Erickson Hall, East Lansing, MI 48824 firstname.lastname@example.org
0-7803-6424-4/00/$10.00 ? 2000 IEEE October 18 - 21, 2000 Kansas City, MO 30 th ASEE/IEEE Frontiers in Education Conference T3A-5
IDENTIFYING AND ASSESSING LEARNING OBJECTIVES
To identify and assess learning objectives for ECE 345, we focused on four constituencies or levels of concern: (a) within the ECE 345 course, (b) client departments, (c) peer institutions and (d) the home department. Below we describe the rationale for these selecting these foci, the specific role of groups within each foci, the type of information needed from these groups and the methods for collecting and analyzing relevant data. Within Course The first constituency for ECE 345 contains the faculty member in charge of the course, teaching assistants (TAs) for the nine laboratory sections and 2 laboratory technicians as well as the students taking it. The emphasis here is on developing a consensus by the individuals responsible for the course about learning objectives for students and the ways to accomplish those objectives. Since ECE 345 had been taught by the same faculty member for at least ten years, an individual now retired, the first step for the new faculty member in charge was to review the current course syllabus and laboratory manual. This review uncovered a large body of relevant content to be covered as well as s ome obsolete experiments and lectures. This review helped the faculty member in charge to decide on an initial content for the revised course. Following ABET guidelines  and recent literature on teaching and learning  , the faculty member then identified an initial set of learning objectives and planned a new instructional format making greater use of student teams and active learning concepts. Since much of the actual teaching and learning process in ECE 345 happens in the laboratory and in interaction between TAs and laboratory technicians with students, training and gaining the cooperation of TAs and technicians proved every bit as important as the initial course preparation by the faculty member. Based on previous work by Colbeck , we developed an interview guide and carried out interviews to identify the types of instructional approaches typically used by home-department faculty and TAs and to identify possible impediments to changing ECE 345. A sample of this interview guide can be found in Appendix A, with the complete guide available at the course web site . To assess student learning outcomes, we modified a self-report instrument developed by Terenzini, et al., designed to measure pedagogical consistency and learning gain . A sampling of the questions from this instrument can be found in Appendix B, with the complete guide available at the course web site . The instrument was used to determine a baseline prior to modifying the course. We have now administered the instrument to students taking the new version of ECE 345 and will compare the two results. We are also in the process of developing a longitudinal follow-up of students who have taken the modified version of ECE 345, focusing on the usefulness of the new course in preparing students for a senior capstone design experience. In all cases, the instrument focuses on effects previously identified as desirable learning outcomes, including experience with design, problem solving, communication skills, and teamwork. Finally, the faculty member responsible for the overall supervision of ECE 345 developed a set of instruments for monitoring ongoing progress and promoting continuous improvement. These instruments are all available at the course web site. One instrument, which is administered twice during the semester, focuses on the students’ assessment of the quality of TA instruction, technician support for the laboratory, the ECE 345 laboratory experiments and the strength of the linkages between the lecture and laboratory portions of the course. A second instrument is used to receive feedback from laboratory TAs on a weekly basis regarding specific experiments. TAs complete a final assessment instrument at the end of the semester to assess the overall the overall laboratory experience. Client Departments Unlike courses in the major, the learning objectives for service courses have typically been less dependent on the needs of students in client departments. The first step was to determine whether or not various client departments required ECE 345 and where the course fit into their degree programs. Once we completed this curriculum review, we interviewed relevant department chairs, curriculum committees, and faculty who taught courses that presumably relied on the material taught in ECE 345 to gather recommendations for ECE 345 learning objectives. We then sent a survey to department chairs to review a proposal for modifying ECE 345 in light of these findings. The survey indicated substantial agreement about the need to reform the course but criticism of any effort to expand the number of credits required for it. Peer Institutions Most peer institutions offer some form of ECE 345. We identified a set of peer programs and surveyed them about their equivalent course. We focused on content, pedagogical approach, and number of credits. Although we found that the ECE 345 equivalent in most peer institutions required more hours and credits than the MSU version, this information was not sufficient to convince MSU client departments to support expanding the credits allocated to ECE 345. Home Department One key to institutionalization is the support for the home department. We interviewed a variety and the department chair in Electrical and Engineering and found little interest among reform by of faculty Computer faculty in
0-7803-6424-4/00/$10.00 ? 2000 IEEE October 18 - 21, 2000 Kansas City, MO 30 th ASEE/IEEE Frontiers in Education Conference T3A-6
helping teach ECE 345. The reluctance was a function of the limited perceived value of a service course for ECE faculty whose promotion, tenure and merit raises was more directly affected by teaching courses for ECE students. We also worked with the ECE Curriculum Committee whose support for reform will be crucial in the long run. However, this home-department curriculum committee was reluctant to endorse any proposed changes in the course until it received formal feedback on these proposed changes from the other departments served by the course.
PROGRESS TO DATE IN REFORMING THE COURSE
The process described in the previous section began in the spring of 1998 and continued throughout the 1998-99 academic year. In the summer of 1999, a specific plan was developed to help bring about systemic reform in ECE 345. Outlined below is some of the progress made toward this end. Revising the Learning Objectives and Course Model On the basis of the feedback received, a revised course syllabus was developed and distributed to the various constituent groups for comment. The current and proposed course syllabi are available for viewing at the course web site . The principal proposed changes are summarized below: ? Increase the number of lecture credits from two to three to reflect the amount of material that the various constituent groups expect to be covered in the course. ? Eliminate differential equations as a prerequisite for the course. ? Strengthen topics related to ac power and electrical safety. ? Add topics related to sensors, actuators, annunciators and their interfaces. ? Add lectures dealing with the application of electric circuits, electronic circuits and embedded computers to the real-time monitoring and control applications. ? Add lectures illustrating linkages between topics learned in ECE 345 and advanced courses within the students’ majors; e.g., the capstone engineering design courses. ? Improve upon students’ communication skills, teaming skills and problem-solving skills. Revising the Laboratory Manual and Experiments
PROCESS USED TO ESTABLISH, REVIEW AND REVISE COURSE LEARNING OBJECTIVES
Before discussing specific course learning objectives with the faculty in other engineering departments, tentative lists of course goals and course topics were developed. It was thought that the points covered on these lists would serve as discussion points for eventually coming to an agreement on a set of course learning objectives and course topics. The list of course goals was: ? convey core knowledge related to electrical engineering fundamentals; ? convey essential information related to electrical safety; ? provide some solid ideas for incorporating EE topics into senior projects for non-EE majors; ? strengthen math skills through engineering applications; ? enhance problem-solving skills. Some possible EE topics that would need to be covered in the context of their application in engineering system design, analysis and testing might include the following: ? power sources and signal sources; ? circuit-analysis fundamentals; ? sensors, actuators, actuators and their interfaces; ? signal processing (e.g., low-pass and high-pass filters); ? test and measurement equipment; ? capstone application—e.g., a simple communication system, digital traffic light controller or an automated control system involving simple sensors and actuators.
Feedback received from former students in the course indicated that the greatest student learning was taking place in the laboratory portion of the course. In retrospect, this is not surprising since the laboratory portion of the course provided students with an opportunity for hands-on learning experience . Additional feedback received from students With these lists in hand, each department chair in the and teaching assistants identified a number of shortcomings College of Engineering was interviewed to determine the in the laboratory manual and laboratory experiments that if chair’s perspective on the present status and future needs of overcome would further enhance the learning experience of ECE 345 in the context of the particular department’s students in the laboratory portion of the course. This undergraduate academic programs. The department chairs feedback was used as a basis for making the following were also asked to identify the chair of the department’s changes in the laboratory manual and experiments: curriculum committee. Each curriculum committee chair ? The laboratory manual was placed on a word processor was then interviewed in a similar manner. Moreover, at the for the first time. This included all figures, tables, time of these interviews, a request was made to place a equations and text. This action would facilitate future discussion of ECE 345 on the agenda for future curriculum continuous quality improvement in the manual and committee meetings Six of the seven departments accepted experiments in the future since it would then be easier to this offer. These meetings focused on receiving feedback on update the material. the current course and discussing possible strategies for ? Teaching assistants formally evaluated the laboratory improving the course. manual and experiment on a weekly basis (see 0-7803-6424-4/00/$10.00 ? 2000 IEEE October 18 - 21, 2000 Kansas City, MO 30 th ASEE/IEEE Frontiers in Education Conference T3A-7
Appendix C) and the manual and experiments as a whole at the end of the semester (see Appendix D). Students in the class evaluated the manual and their experiences in the laboratory twice during the semester (see Appendix E). Feedback received from former students in the class, current students in the class and teaching assistants led to a first round set of revisions to the laboratory experiments and the laboratory manual. All of these changes were consistent with the earlier course model and course syllabus; however, they were intended to overcome some of the shortcomings in the previous delivery of the course topics. New laboratory test equipment—i.e., PC-based oscilloscopes and digital waveform generators—were purchased which enabled students to gain experience with modern instruments that could be integrated into an automated laboratory instrumentation system and could be used to obtain digital output that could be used for technical reports, etc. The laboratory manual and laboratory exercises were revised to reflect the capabilities of this new test equipment. the worst teachers or poorest researchers are assigned then finding faculty volunteers to teach a revised course is problematic. Teaching a service course contains disincentives for faculty members and for the department chair. Students from other majors seldom end up enrolling as graduate students in the home department, a disincentive for the chair, and faculty teaching service courses wonder about their value in promotion and tenure relative to teaching advanced courses in the major, a disincentive for the faculty. Finding a way to give credit toward promotion and tenure is a crucial part of encouraging faculty involvement in reforming engineering service courses. One mechanism is to give substantial weight to obtaining externally funded grants focused on teaching and learning. Faculty understand that the investment a department makes in its service courses—the dollars and support assigned to them—reflect the value the department places on thee courses. Involvement by Other Engineering Faculty Engineering faculty and curriculum committees from departments other than the department with “primary responsibility” for engineering service course have traditionally not been actively involved with the teaching, assessment or reform of engineering service courses. The prevailing attitude appears to have been that “the faculty in the primary department know best and will do what is best to serve the needs of the engineering students taking the particular engineering service course.” Little thought or effort has traditionally been given by these faculty to explore ways of improving an engineering student’s overall learning experience by linking lessons learned within a particular engineering service course with advanced engineering topics within the student’s major field of study. Moreover, while the employers of engineering graduates have become much more interested in the engineering graduate’s ability to function on interdisciplinary teams, there has been little effort on the part of the faculty to determine how to best foster these teaming experiences in the undergraduate learning experience. Opportunities for Interdisciplinary Teaming One key to improving student learning at the undergraduate level appears to hinge upon recognition of the fact that engineering service courses might well serve as an important catalyst for interdisciplinary teaming within a student’s overall undergraduate experience. For example, within the ECE 345 lecture and laboratory learning environment, engineering students from a diverse set of technical backgrounds would quite naturally have the opportunity to interact with each other on multi-disciplinary teams. Moreover, one specific learning objective of the course itself
The overarching goal of this educational research project has been two-fold . First, we are seeking ways to improve the quality of the undergraduate student learning experience through reform of the engineering service courses. And, second, we are seeking ways to ensure systemic change by institutionalizing these reforms in the curriculum. We have revised the course to account for the needs of constituent departments, changed the pedagogical approach to encourage active student learning, revised the laboratory manual and experience and arranged to use new assessment tools. However, we have not implemented these changes on an ongoing basis because of external factors affecting systemic reform. Home Department Faculty Involvement Systemic reform of engineering service courses, such as ECE 345, requires the active involvement by homedepartment faculty in all aspects of the teaching, evaluation and evolution of the course materials. Perhaps the most significant finding to date relates to the faculty’s lack of interest in becoming actively involved in service courses such as ECE 345 from either teaching or scholarship perspective. Through interviews with the faculty in the Department of Electrical and Computer Engineering (ECE), we have been able to identify the following reasons for this reluctance on the part of home-department faculty to become involved with these types of engineering courses. ? The history of service courses influences the value that faculty place on them. If assignment to teach a service course is seen as a punishment or as the course where
0-7803-6424-4/00/$10.00 ? 2000 IEEE October 18 - 21, 2000 Kansas City, MO 30 th ASEE/IEEE Frontiers in Education Conference T3A-8
might include improving upon the ability of non-electrical and non-computer engineers to function on engineering teams that contain electrical and computer engineering majors. Once this linkage was understood, we came to recognize that these types of teaming experiences could extend beyond the service course itself and into major engineering design experiences that involved multidisciplinary teams. For example, mechanical engineering students often need to incorporate electronic sensors, actuators, annunciators, signal processing, data logging, computers and electronic instrumentation into their senior design projects. The Next Steps in Achieving Systemic Reform In our quest to establish learning objectives and assess outcomes in engineering service courses, we have come to learn that these courses should not be viewed in isolation but rather need to be placed in the context of the engineering student’s overall undergraduate learning experience. We have identified several short-range tasks that will help bring this about in the long term. ? We have established a task force comprised of ECE faculty and faculty from key departments currently served by ECE 345. This task force will assess the current status of ECE 345 and make recommendations for future changes in the course and its linkages to the academic programs served by the course. ? We will continue to seek ways to achieve systemic reform in engineering service courses, such as ECE 345, by identifying and addressing the institutional barriers that have discouraged department faculty from becoming more actively involved in the teaching and scholarship associated with these courses. ? We will continue to seek ways to develop stronger linkages between engineering service courses, such as ECE 345, and advanced courses within the engineering student’s major.
 Fisher, P. D., Assessment Process at a Large Institution, Proc. of the 1998 ASEE Annual Meeting and Exposition, Seattle, WA, June 28July 1, 1998, CD-ROM. Fisher, P.D., Fairweather, J.S., Transforming Engineering Service Courses, Proc. of the 1999 Frontiers in Education Conference, San Juan, PR, Nov. 10-14, 1999, CD-ROM. Fisher, P.D., Fairweather, J.S., Rover, D.T., Haston, L.A., Linking Engineering Service Courses with Engineering Design , Proc. of the 2000 ASEE Annual Conference, St. Louis, MO, June 18-21, 2000, CD-ROM. Fisher, P.D., et al., ASEE 2000 Workshop: Linking Engineering Service Courses with Engineering Design, Web Site for the 2000 ASEE Annual Conference Workshop, St. Louis, MO, June 18-21, 2000, URL: http://www.egr.msu.edu/~fisher/asee2000_work_shop. Course Web Site for Electronic Instrumentation Systems (ECE 345), Michigan State University, URL: http://www.egr.msu.edu/ece/courses/ece345.htm. Terenzini, P., et al., ECSEL Year 7 Evalu ation Report, University Park, PA: Center for the Study of Higher Education, Penn State University, 1996. Moore, K., et al., Best Practices for Reform in Undergraduate Education in Science, Math, Engineering, & Technology: A Knowledge Framework, East Lansing, MI: Center for the Study of Advanced Learning Systems, Michigan State University, 2000.
 Angelo, T., and Cross, P., Classroom Assessment Techniques: Handbook for College Teachers, 2nd . Ed ., San Francisco: Jossey-Bass, 1990.  Weimer, M., Improving College Teaching: Strategies for Developing Instructional Effectiveness, San Francisco: Jossey-Bass, 1990.  Colbeck, C., Merging in a Seamless blend: How Faculty Integrate Teaching and Research, Journal of Higher Education, 69, pp. 647671.  Goodsell, A., Maher, M., and Tinto, V. Collaborative Learning: A Sourcebook for Higher Education, University Park, PA: National Center on Postsecondary Teaching, Learning and Assessment, 1994.
APPENDIX A: FACULTY & TA INTERVIEW GUIDE
Teaching Methods 1. 2. 3. 4. 5. 6. What do you consider the most important content and skills for students to learn in this class? Describe the teaching methods that you use/have used/will use in this class. Tell me about any new or different teaching methods you may have tried in the class. What kinds of teaching methods do most of your engineering colleagues use in their undergraduate courses? Describe the evaluation and grading methods you use in this class. About your teaching: a. What are your particular strengths in working with undergraduates? b. How would you like to improve the ways you work with undergraduates?
The authors would like to acknowledge the assistance provided by Mr. Nathan Robinson and Mr. Eric Warmbier in collecting the data for the ECE 345 benchmarking process. Also, the authors would like to acknowledge the important comments made by the two other principal investigators involved in the project: Dr. Susan Masten, Department of Civil and Environmental Engineering, and Dr. Jon Sticklen, Department of Computer Science and Engineering.
 Criteria for Accrediting Programs in Engineering in the United States: Effective for Evaluations During the 1998-99 Accreditation Cycle, Engineering Accreditation Commission, Accreditation Board for Engineering and Technology, Baltimore, MD, pp. 5-7. ABET Engineering Criteria 2000, URL: http://www.abet.ba. md.us/EAC/eac2000.html.
0-7803-6424-4/00/$10.00 ? 2000 IEEE October 18 - 21, 2000 Kansas City, MO 30 th ASEE/IEEE Frontiers in Education Conference T3A-9
c. 7. How—if at all—has your approach to working with undergraduates to enhance their learning changed over the past 5 to 10 years? How do you balance undergraduate teaching with your other faculty responsibilities? Curricular Impact 8. What needs to happen to integrate the new activities and focus of this course with other courses in the curriculum? Broad Educational Goals 7. Use established criteria to evaluate and prioritize solutions? 8. Understand that a problem may have multiple solutions? 9. Develop ways to resolve conflict and reach agreement in a group? 10. Identify the constraints on the practical application of an idea?
In your view, what are the most important thing students majoring in [this discipline] should learn by the time they graduate? In particular, I am interested in content knowledge, skills, integration or application of knowledge, and attitudes toward future learning. 10. What is the most effective way that faculty can contribute to student learning in each of these areas? Departmental Policies 11. How do formal policies of your department facilitate or hinder your ability to promote undergraduate learning? I am interested particularly in workload pollicies, teaching assignments, faculty involvement in decisionmaking, and rewards. 12. What will it take to institutionalize the teaching and innovations started in this course in your department?
APPENDIX B: STUDENT CLASSROOM LEARNING EXPERIENCES LEARNING OUTCOMES CONSISTENT WITH REFORM OBJECTIVES IN ENGINEERING EDUCATION
Instructions: Please respond to each of the following statements using the following code: 1=None; 2=Slight; 3=Moderate; 4=A Great Deal How much progress have you made, because of this course, in your: 1. 2. 3. 4. 5. 6. Understanding of what engineers “do” in industry or as faculty? Knowledge and understanding of the language of design in engineering? Knowledge and understanding of the process of design in engineering? Your ability to “do” design? Apply an abstract concept or idea to a real problem or situation? Clearly describe a problem in writing?
0-7803-6424-4/00/$10.00 ? 2000 IEEE October 18 - 21, 2000 Kansas City, MO 30 th ASEE/IEEE Frontiers in Education Conference T3A-10