Significant Results
Nine Principles of Good Practice for Assessing Student Learning
by the American Association of Higher Education (AAHE)
Assessing Student Learning Worksheet
Using Assessment for Student Remediation and Program Improvement
by Dennis George, Assistant Vice President for Institutional Effectiveness
and Retta Poe, Associate Dean, College of Education and Behavioral Sciences
SITE and Significant Results by Sally Kuhlenschmidt,
FaCET Director /Psychology, WKU
To Teach the Sea by Scott Bonham, Assistant Professor,
Physics and Astronomy
Significant Results: Ten Items for the Bottom Line
by Ken Kuehn, Professor, Geography and Geology
Top Five Ways to Bring About Significant Results from your
Instruction by Sally Kuhlenschmidt, FaCET Director /Psychology, WKU
Technology-Based Learning and Its Benefits for Adult
Learning by Jim Berger,Assistant Professor, Special Instructional
Programs
Top Five Ways to Measure Significant Results by
Sally Kuhlenschmidt, FaCET Director /Psychology, WKU
Student Assessment Resources by Nancy Givens,
Instructional Coordinator, Faculty Center for Excellence in Teaching
Introduction
We are pleased to offer you the fifth in our series of booklets on the six standards of scholarship as it applies to teaching. This edition focuses on the meaning of “Significant Results” as it applies to teaching. A quote from the referenced source provides background:
“All works of scholarship, be they discovery, integration, application, or teaching, involve a common sequence of unfolding stages. We have found that when people praise a work of scholarship, they usually mean that the project in question shows that it has been guided by these qualitative standards: 1) Clear Goals, 2) Adequate Preparation, 3) Appropriate Methods, 4) Effective Presentation, 5) Significant Results, and 6) Reflective Critique.”
This volume explores a range of topics from assessment to determine Significant Results at program and course levels to philosophical considerations.
“Developing teaching skills is a process, one that can engage a person
for an entire career. The standards proposed by Dr. Glassick and others suggest
a route we can follow to sustain quality in our teaching outcomes. This series
of booklets may serve as starting points for thought and elaboration.”
- Sally Kuhlenschmidt, Director FaCET/Psychology
Additional copies of this booklet are available if you call 270-745-6508 or online at http://www.wku.edu/teaching/booklets/goals.htm.
Nine Principles of Good Practice for Assessing Student Learning
By American Association for Higher Education (1991).
1. The assessment of student learning begins with educational values. Assessment is not an end in itself but a vehicle for educational improvement. Its effective practice, then, begins with and enacts a vision of the kinds of learning we most value for students and strive to help them achieve. Educational values should drive not only what we choose to assess but also how we do so. Where questions about educational mission and values are skipped over, assessment threatens to be an exercise in measuring what's easy, rather than a process of improving what we really care about.
2. Assessment is most effective when it reflects an understanding of learning as multidimensional, integrated, and revealed in performance over time. Learning is a complex process. It entails not only what students know but what they can do with what they know; it involves not only knowledge and abilities but values, attitudes, and habits of mind that affect both academic success and performance beyond the classroom. Assessment should reflect these understandings by employing a diverse array of methods including those that call for actual performance, using them over time so as to reveal change, growth, and increasing degrees of integration. Such an approach aims for a more complete and accurate picture of learning, and therefore firmer bases for improving our students' educational experience.
3. Assessment works best when the programs it seeks to improve have clear, explicitly stated purposes. Assessment is a goal-oriented process. It entails comparing educational performance with educational purposes and expectations–these derived from the institution's mission, from faculty intentions in program and course design, and from knowledge of students' own goals. Where program purposes lack specificity or agreement, assessment as a process pushes a campus toward clarity about where to aim and what standards to apply; assessment also prompts attention to where and how program goals will be taught and learned. Clear, shared, implementable goals are the cornerstone for assessment that is focused and useful.
4. Assessment requires attention to outcomes but also and equally to the experiences that lead to those outcomes. Information about outcomes is of high importance; where students "end up" matters greatly. But to improve outcomes, we need to know about student experience along the way–about the curricula, teaching, and kind of student effort that lead to particular outcomes. Assessment can help understand which students learn best under what conditions; with such knowledge comes the capacity to improve the whole of their learning.
5. Assessment works best when it is ongoing, not episodic. Assessment is a process whose power is cumulative. Though isolated, "one-shot" assessment can be better than none, improvement is best fostered when assessment entails a linked series of activities undertaken over time. This may mean tracking the progress of individual students, or of cohorts of students; it may mean collecting the same examples of student performance or using the same instrument semester after semester. The point is to monitor progress toward intended goals in a spirit of continuous improvement. Along the way, the assessment process itself should be evaluated and refined in light of emerging insights.
6. Assessment fosters wider improvement when representatives from across the educational community are involved. Student learning is a campus-wide responsibility, and assessment is a way of enacting that responsibility. Thus, while assessment efforts may start small, the aim over time is to involve people from across the educational community. Faculty play an especially important role, but assessment's questions can't be fully addressed without participation by student-affairs educators, librarians, administrators, and students. Assessment may also involve individuals from beyond the campus (alumni/ae, trustees, employers) whose experience can enrich the sense of appropriate aims and standards for learning. Thus, understood, assessment is not a task for small groups of experts but a collaborative activity; its aim is wider, better-informed attention to student learning by all parties with a stake in its improvement.
7. Assessment makes a difference when it begins with issues of use and illuminates questions that people really care about. Assessment recognizes the value of information in the process of improvement. But to be useful, information must be connected to issues or questions that people really care about. This implies assessment approaches that produce evidence that relevant parties will find credible, suggestive, and applicable to decisions that need to be made. It means thinking in advance about how the information will be used, and by whom. The point of assessment is not to gather data and return "results"; it is a process that starts with the questions of decision-makers, that involves them in the gathering and interpreting of data, and that informs and helps guide continuous improvement.
8. Assessment is most likely to lead to improvement when it is part of a larger set of conditions that promote change. Assessment alone changes little. Its greatest contribution comes on campuses where the quality of teaching and learning is visibly valued and worked at. On such campuses, the push to improve educational performance is a visible and primary goal of leadership; improving the quality of undergraduate education is central to the institution's planning, budgeting, and personnel decisions. On such campuses, information about learning outcomes is seen as an integral part of decision making, and avidly sought.
9. Through assessment, educators meet responsibilities to students and to the public. There is a compelling public stake in education. As educators, we have a responsibility to the publics that support or depend on us to provide information about the ways in which our students meet goals and expectations. But that responsibility goes beyond the reporting of such information; our deeper obligation-to ourselves, our students, and society-is to improve. Those to whom educators are accountable have a corresponding obligation to support such attempts at improvement.
The Authors
Alexander W. Astin, University of California at Los Angeles; Trudy W. Banta,
Indiana University-Purdue University at Indianapolis; K. Patricia Cross, University
of California, Berkeley; Elaine El-Khawas, American Council on Education;
Peter T. Ewell, National Center for Higher Education Management Systems; Pat
Hutchings, American Association for Higher Education; Theodore J. Marchese,
American Association for Higher Education; Kay M. McClenney, Education Commission
of the States; Marcia Mentkowski, Alverno College; Margaret A. Miller, State
Council of Higher Education for Virginia; E. Thomas Moran, State University
of New York; Barbara D. Wright, University of Connecticut.
This document was developed under the auspices of the AAHE Assessment Forum,
a project of the American Association for Higher Education, with support from
the Fund for the Improvement of Postsecondary Education. It builds on earlier
efforts, by campuses and other groups, to articulate guidelines for assessment's
practice; its intent is to synthesize important work already done and to invite
further statements about the responsible and effective conduct of assessment.
Reproduced by permission of the publisher. Copyright © 1991, The American
Association for Higher Education and Copyright © 2005, by Stylus Publishing,
LLC
Assessing Student Learning Worksheet
Assessment is a systematic process of gathering, interpreting and using information about student learning. As such, it can be an important tool for educational improvement. Carefully designed, assessments can positively affect how students approach their study and the quality of their learning.
Using the Nine Principles of Good Practice for Assessing Student Learning (page 3), how would you rate your courses or programs, on a scale of 1 to 5, where ‘1’ is ‘never consider the principle’ and ‘5’ is ‘achieve the principle’, for each of the identified practices?
| GOOD PRACTICE | SELF-RATING |
2. Assessments reflect an understanding of learning as multidimensional, integrated, and revealed in performance over time. 1 2 3 4 5
3. Courses or programs you assess have clear, explicitly stated purposes. 1 2 3 4 5
4. Assessments include evaluation of student experiences that lead to desired outcomes. 1 2 3 4 5
5. Assessments of student learning are ongoing, not episodic. 1 2 3 4 5
6. Assessments foster wider improvement by involving appropriate representatives from across the educational community, (e.g.,librarians, Student Affairs staff, alumni) 1 2 3 4 5
7. Assessments illuminate questions that people really care about. 1 2 3 4 5
8. Assessments are part of a process that promotes improvement in learning. 1 2 3 4 5
9. Assessments meet responsibilities to students and the public. 1 2 3 4
5
Top
of Page
Using Assessment for Student Remediation and Program Improvement
by Dennis K. George, Assistant Vice President for Academic Affairs
and Provost’s Office, and Retta E. Poe, Associate Dean College of Education
and Behavioral Sciences, WKU
Recently, Western Kentucky University initiated a formal procedure for organizing and documenting assessment activities for all academic programs. Although assessment in various forms has been taking place in academic programs and support units across campus for many years, we have not had a common organizational format or a formal system for reporting our efforts. Our recent emphasis on assessment activities signifies our awareness of the importance of consistent implementation and documentation of assessment. The hope is that improvements in these areas will lead not only to increased institutional effectiveness but also to a higher quality of reports and greater ease in the preparation of reports, such as those required for accreditation and for our internal academic program review process.
Faculty are already familiar with one important type of assessment: assessment of students. Student assessment is similar to program assessment in some ways, but it is important to understand the differences between them. The unit of measurement is the same in both cases: the student or some product generated by the student, such as a capstone project or portfolio. Also, many of the assessment tools for both student assessment and program assessment are the same or similar. However, these processes differ in the overall purpose of the assessment and in the primary utilization of the results.
Faculty employ student assessment tools to determine the competency of each student who completes the program. That is, faculty attempt to measure whether program graduates meet the faculty’s expectations with respect to the students’ skills or knowledge of subject matter in the field of study. Every graduate must be certified through some demonstration that involves more than simply completing a certain number of courses. In principle this demonstration “guarantees” that all graduates have achieved the required competencies at the specified levels of mastery, and those who fail to measure up may be provided tutoring or expected to complete some other remediation process.1
WKU implemented a formal program of student assessment in the mid-1990s by establishing the requirement for a “Senior Exam” for all graduating seniors. The following statement from the 2001 – 2003 Catalog articulates the university’s expectation:
Senior Exam—Effective with the Fall 1996 entering freshmen, all graduating seniors will be required to show evidence of knowledge in their field. Each program area will require students to provide evidence that standards have been met through portfolio, examination or other culminating experiences.
The implication of this policy statement was that the results of the senior assessment were to be used to evaluate the competency of individual students. Although it was not specified in the Senior Exam policy statement, it was presumed that some type of student-focused remedial action would occur if students were found not to have met prescribed standards.
Individual programs addressed this requirement in various ways, including comprehensive examinations, performance appraisals, juried reviews, portfolios, capstone courses, etc. However, although we established a substantial history of carrying out comprehensive student assessment, there was no system for the formal reporting of the results, nor were program faculty required to use the results for any purpose beyond ensuring individual student competency. Having recognized the need for program assessment information to enable judgments about our success in achieving program expectations for students, in Fall 2002 we instituted a new requirement that faculty in each program annually document program assessment efforts and results.
Program assessment is a requirement of our regional accrediting body, the Commission on Colleges of the Southern Association of Colleges and Schools (SACS). SACS Comprehensive Standard 3.3, Institutional Effectiveness, states, “The institution identifies expected outcomes for its educational programs and its administrative and educational support services; assesses whether it achieves these outcomes; and provides evidence of improvement based on analysis of those results.” Thus, we are specifically required to develop “outcomes” or objectives for all academic and administrative and educational support programs. The identification of some means to assess the achievement of these outcomes is also necessary, as well as the specification of some indicator of successful achievement. Most importantly, the expected effect is improvement in the programs based on the analysis of the results.
What, then, are the differences between student assessment and program assessment? Whereas student assessment focuses on documenting the competency of individual students, program assessment is undertaken for the purposes of judging and ultimately improving overall instructional performance. Faculty establish a set of student learning outcomes (defined as what students are expected to know, think, or be able to do by the time of graduation2), the attainment of which is assessed through the collection and use of aggregated data. If these outcomes are not achieved, the focus of remediation is the program itself through actions such as changes in course content or sequence, modification of advising processes, creation of new courses, etc.
Given that we must document both student and program assessment, it would save us some work if we were able to use the same assessment tools to accomplish both worthy efforts. Luckily, this is possible to do in many situations, simply by using aggregated data from individual student assessments (e.g., Senior Exams) for the purpose of program assessment.
Suppose, for example, that the student assessment for the Bachelor of Science in Environmental Health and Safety (EHS) program is a written examination that is given for a grade in a required senior capstone course. EHS program faculty write exam questions to assess student knowledge with respect to five desired outcomes: (1) Application of scientific and mathematical principles, (2) Personal protective equation utilization, (3) Monitoring of physical and chemical exposures, (4) Harmful effects of environmental agents, and (5) Design of local exhaust ventilation systems. The exam consists of 50 multiple choice questions, each of which is worth two points, for a total of 100 points, and students who fail to earn at least 70% of the available points then receive tutoring in areas of weakness (this is the criterion for defining success for students).
Now, could the EHS faculty use the results of this same exam to focus on improvement of the overall academic program? The answer is definitely “yes,” provided the faculty are able to match exam questions to specific program outcomes. Suppose that questions 1-10 measure achievement of outcome 1 (Application), questions 11-20 measure Outcome 2 (PPE), questions 21-30 measure Outcome 3 (Monitoring), questions 31-40 measure Outcome 4 (Agents) and questions 41-50 measure Outcome 5 (LEV). The faculty could then establish criteria for success for the program as follows: The average grade on the Senior Exam will be no less than 70%, and on no individual outcome area will the average score be less than 14 points.
As an illustration, suppose that 10 students are in this particular course
section, and they receive the following scores on the exam:
| Students | Outcome 1 Application |
Outcome 2 PPE | Outcome 3 Monitoring | Outcome 4 Agents | Outcome 5 LEV | Total | Grade |
| Student 1 | 20 |
16 |
16 |
20 |
12 |
84 |
B |
| Student 2 | 20 |
20 |
20 |
12 |
12 |
84 |
B |
| Student 3 | 16 |
12 |
16 |
8 |
8 |
60 |
D |
| Student 4 | 20 |
20 |
20 |
16 |
16 |
92 |
A |
| Student 5 | 20 |
16 |
16 |
12 |
12 |
76 |
C |
| Student 6 | 16 |
16 |
12 |
16 |
12 |
72 |
C |
| Student 7 | 12 |
12 |
12 |
12 |
8 |
56 |
F |
| Student 8 | 20 |
16 |
16 |
16 |
12 |
80 |
B |
| Student 9 | 20 |
20 |
16 |
20 |
12 |
88 |
B |
| Student 10 | 16 |
20 |
12 |
16 |
12 |
76 |
C |
| Average | 18 |
17 |
16 |
15 |
12 |
77 |
Using the exam results for student assessment, the faculty would probably conclude that while most students performed satisfactorily, Students 3 and 7, who received scores of 60 and 56, respectively, need individualized tutoring to address deficiencies.
However, note that aggregating the individual student data (i.e. looking down the columns and examining the averages on each section) in order to use the data for program assessment leads to the conclusion that the program is not doing a good job helping students achieve Outcome 5, Local Exhaust Ventilation design. Students consistently scored lower on the questions relating to this outcome than on any other. In fact, the average score was 12 points, less than the established criterion for success. Thus, the program assessment data identify for the program faculty an aspect of the instructional program that needs improvement. The actions that might be taken include changing course content to emphasize LEV in certain courses, changing course sequencing to better prepare students to understand LEV concepts, creating a new course solely dedicated to LEV, reviewing the importance of LEV as a program outcome, and evaluating the validity of the exam for assessing LEV.
The point of this illustration is to show that with some consideration beforehand, faculty can sometimes use comprehensive exam data for program assessment. The key is creating exam questions tied to program outcomes and looking carefully at the subscale scores.
While the above example details the use of a comprehensive exam for program assessment, faculty may take a similar approach with other typical student assessments, such as essays, portfolios, juried performances, oral presentations, theses, internship evaluations, etc. Each of these tools can be utilized for program assessment as well as student assessment, as long as performance on the tool relates to one or more of the program outcomes. Of course, the advantage of this approach is that it enables faculty to make more efficient use of student work, using it once to assess individual students and then again to assess various aspects of the overall program.
If you’d like more information on how to use student assessment results in program assessment, feel free to contact Dennis George (dennis.george@wku.edu) or Retta Poe (retta.poe@wku.edu).
References:
1. Ewell, Peter T., Accreditation and Student Learning Outcomes: A Proposed
Point of Departure, Council for Higher Education Accreditation. Washington,
D.C. (September 2001).
2. Nichols, James O. and Nichols, Karen W. The Departmental Guide and
Record Book for Student Outcomes Assessment and Institutional Effectiveness,
3rd Edition. Agathon Press, New York (2000).
SITE and Significant
Result
by Sally Kuhlenschmidt, Director Faculty Center
for Excellence in Teaching, WKU, Fall 2005
In each of this series of booklets I have examined how the booklet focus may impact student ratings. One interpretation of the meaning of ‘significant results’ is ‘what the student learns from the course.’ How might student learning relate to the various student rating items?
My instructor displays a clear understanding of course topics. It seems obvious that the odds of student learning should increase to the extent that the instructor understands course material. If the instructor is not well versed in the subject matter for the course he or she will have difficulty answering questions, structuring material, and anticipating problems and helping students resolve them. But simply understanding isn’t enough if the instructor cannot convey that understanding effectively and clearly, that is, if the instructor cannot share or display that information. The instructor interested in significant results and in performing ably on this item will work on both content knowledge and the effectiveness of how his information is displayed—Are materials organized for presentation? Are illustrations effective? Do activities illustrate key concepts? Do students understand what the objective of the lesson is?
My instructor displays interest in teaching this class. How can students value the information if the teacher doesn’t seem to care? Enthusiasm and motivation are contagious. If the teacher is not motivated to teach a class it will be an obstacle to student motivation and will reduce learning. Part of success in teaching for learning is removing or reducing obstacles. To optimize the significant result of student learning, the teacher has to value that learning and the students and show it.
My instructor is well prepared for class. Perhaps more challenging for some teachers is being well prepared for class given the demands on instructors’ time. Nevertheless, being unprepared undermines student learning. One important role of the teacher is to prepare students for learning by providing objectives and a mental map of where the course is heading. A mind that is prepared for the information is a mind that is able to absorb more and use that information in learning. Teacher preparation is also a sign of respect for the student’s time. Lack of preparation conveys that you do not value the time set aside for class. Being prepared can also mean adding special touches and examples that enliven material.
My instructor is actively helpful. A teacher who is actively helpful to students can also enhance learning. Helpful may simply be listening as a student figures out the solution. It may be expressing sympathy. Of course it may also be breaking a problem into smaller parts and giving suggestions for how to resolve a difficulty. It is possible to think you are helping when you may actually hinder student learning because you do not have them ‘construct’ the learning on their own. Finding the right balance between independent functioning and teacher intervention must be done anew each semester.
Performance measures (exams, assignments, etc.) are well constructed. If measures are poorly designed then the teacher won’t have an accurate measure of how students are learning (or aren’t). Even if you know the test is well designed, if students perceive it as arbitrary or capricious, they may not respect the results or use them to improve. If designed well, measurement tools may be used to enhance student learning.
My instructor treats me fairly with regard to race, age, sex, religion, national origin, disability, and sexual orientation. If the instructor does not treat students equitably, strong emotions may surface, then it will be difficult to achieve significant results in student learning. If one person is singled out for unfair treatment, the others in the class are aware that capriciousness is possible and will be distanced from the material.
Overall, my instructor is effective. What is effectiveness? It is helping students create their learning.
by Scott Bonham, Assistant Professor, Physics and Astronomy
SUMMARY: If a fisherman is not catching many fish, he could just blame the fish. On the other hand, he could study the fish’s environment, get maps of the ocean floor and utilize radar to find where they are. After all, fish and fishermen live in very different worlds, and most fish are not going to seek to cross that gap. Teaching is a little like that. There can be significant differences in what consists of ‘learning.’ There are certain places where confusions tend to breed. Our students are moving targets like and unlike other years, and the language gap can be significant. Applying the study of teaching and learning in our classrooms may not change our students, but it can help make sure our instruction reaches students on the other side of the gap between the two worlds.
If physics teachers are together long enough, sooner or latter the topic will
turn to students that “didn’t get it.” No matter how brilliant
the lecture, how simple the explanation, or how many times the table was banged,
it simply didn’t register. It wasn’t just one student who failed
to grasp it—the pearls of wisdom just flowed in and out of most of the
ears in the class. It sometimes seems like trying to teach the sea—lots
of movement, but, at the end of the day it is still the same. This frustration
drives some instructors to rush back and revise their notes, others to dive
into new technology and multimedia, and others to throw in the towel and blame
it all on “lazy,” “unprepared,” or worse, students.
The students, of course, murmur about hard tests, incomprehensible lectures,
and impossible material. Each semester rolls in and out while students flow
through the halls until professor’s heads are capped with white. The
gulf between what professors teach and what students learn is unchanged. The
crabs, clams and crowds come and go, but the sea remains.
If you have ever tried to get the attention of someone who is swimming underwater, you know that is a nearly impossible task. As soon as the ears submerge beneath the water, all of the noise and confusion from above dies down to a soft murmur. Olympic synchronized swimmers have to use special underwater speakers to not lose their place when they dive under. The person in the water is not necessarily deaf or ignoring you (though perhaps that is why Johnny dove underwater as his mother approached). It is a simple physical fact that when sound travels from a low density medium (air) to a high density one (water) most of the sound gets reflected. Very little sound gets through, and what does is distorted. Air is a different medium than water, and the boundary between them can greatly interfere with communication.
It might be comforting to attribute the gulf between what is taught and what is learned to a difference between media. That way one does not need to blame either the professors for being incompetent or the students for being slow. But is there any truth to that? It turns out that there is. At least part of the problem is a difference in intellectual media. There can be a significant difference in expectations, particularly in what it means and takes to learn. There can be a significant difference in foundational understanding, sometimes in subtle but profound ways. Finally, there can be a significant difference in language. It is worth looking at these differences more closely, through which to discover pointers in building “underwater speakers.”
Students and professors often enter the classroom with very different ideas of what is involved in learning the subject. This can be quite significant in introductory physics,1 particularly in what physics knowledge is and the role of students in the learning process. Students tend to see physics more as collections of facts and equations while instructors usually see it as a coherent, systematic understanding of relationships between things. Students are more likely to see their role in the classroom as to receive well-digested knowledge from their instructors, who in turn expect the students to be more active participants, trying different approaches until they understand things. A frequent result of this gulf is students trying to memorize the specific steps in solving specific problems while failing to grasp the reasoning process behind it. This then leaves them to feel that the test consisted of problems completely different from the homework, and leaves the instructor baffled at how the students missed questions “just like” ones already worked.
While different subjects will have their own particular issues, our greater familiarity with our discipline almost invariably engenders a more sophisticated understanding of what it means to “know” a subject. We have also had to become active learners who manage our own learning, something most of our students have not learned to do. We certainly cannot blame our students for less sophisticated understandings of the meaning and process of learning, since it is only through the learning process those understandings and skills can develop. However, unrecognized and unaddressed, these differences can cause real damage. In my discipline there is a temptation to “dumb-down” the tests to the level that the students can memorize their way through. While it may reduce the frustrations, this only re-enforces erroneous perceptions of the subject and squanders an opportunity for deeper learning. A better approach is to make the learning objectives of the course very explicit and ensure that tests and other evaluations are aligned with them. Students can be helped in learning by explicit teaching of metacognitive skills, from discipline specific things such as the steps of a physics problem solving strategy to general things like discovering their own learning style.2
Expectations and learning process, of course, are the salt of the course. A little can make a real difference, but a lot just tastes bad. The focus, of course, is the subject matter. At least in introduc-tory physics, this is where the difference in medium can really knock us down. A naïve view of education sees students as blank slates, ready for the instructor to inscribe with words of wisdom. After eighteen or more years of informal and formal learning, our students are far from blank, and what we can write can be greatly affected by what is already there. I am only half joking when I tell my class that they have been studying physics ever since they started dropping toys from their high chair. The problem is what many of them have learned deeply conflicts with what I want to teach them.
Learning, as most of us would recognize, is not as simple as opening up a student’s head and “dumping” knowledge in. It is more of a construction project, like building a ship in a bottle, where pieces of knowledge are brought in one at a time and assembled in the student’s mind. Unlike computers, where information is stored in discrete values in different parts of its memory, our brains learn by creating and altering connections between nerve cells. The problem is that if I don’t have the hull of my ship constructed properly, the deck won’t fit and I won’t have a place for one of the masts.
Many students come into my class having observed countless sliding blocks, rolling bicycles and almost every other object in life, reasonably concluding that everything will eventually slow to a stop if there isn’t a force on it. Unfortunately, that flatly contradicts one of Isaac Newton’s fundamental principles that everything will, in absence of a force (such as friction) continue to move with an unchanged velocity. This is not a trivial matter, since the relationship between the force on an object and its motion is one of the most fundamental relationships in first semester physics. If a student doesn’t get force and motion right, there is little chance they will master momentum and impulse, or torque and rotation.
The most troubling thing is, however, is that this fundamental conflict can stay well hidden below the surface unless I specifically target it. Their common-sense ideas about motion and force lie at a gut level arising from personal experience, while what I say is propositional knowledge coming from an outside authority. These incompatible ideas seem to be able to peacefully co-exist in different sections of many student minds for a semester or more. Sometimes I have to provoke the conflict, by asking them to predict what will happen before they do an experiment. One class I will give each group a cart with low-friction wheels and a spring scale, and ask them how the cart will move when they maintain a constant force on it. There are a lot of surprised students when they discover that they have to go faster and faster to maintain the constant force, instead of the constant speed they predicted. However, beliefs about the world are not built in one day, so they are certainly not torn down in fifteen minutes. Later in that same class, they do another activity using a computer to graph in real time the force, velocity and acceleration of a cart to reinforce the idea, and follow up the next day with yet another activity, and the day after that with a lecture on the topic. Even so, there are some hard cases that resist all the prying.
The key, of course, is knowing where to fish. Just as a deep-sea fisherman faces a lot more water than fish to be caught, we face a lot more material than what can be satisfactorily hammered in a semester like I described above. Furthermore, the people who hired the fishing guide and his boat don’t want to waste their time discovering all the unproductive places to fish. Fortunately, the modern deep-water fishing guide has a number of tools to find the best places to fish. Although the surface of the sea may be featureless as far as the eye can see, underneath there are rocks and reefs, open water and kelp beds, shallows and depths. The fish come and go, but the unchanging underwater structure influences where they feed, mate and rest. Other people have spent significant time and resources to explore and map the sea floor, as well as study the fish themselves, information that our fishing guide can purchase or even download to narrow the hunt. The professional fishing guide, however, usually goes further than that. Maps and the like provide good general guides, but fish are dynamic creatures that don’t stay in one place. Sonar, and other underwater probes, can locate exactly where they are at a given moment. That way the fishing can focus on the place where the fish actually are, not just where they might be.
In my field I can utilize over two decades of research on student learning of physics that maps out the preconceptions, stumbling blocks, and effective techniques.3 As it turns out, these are fairly stable structures; year after year students come through my class with the same incorrect common-sense ideas about physics. Individual students may vary widely, but class averages are nearly the same from semester to semester. Not only does this give pointers to where to fish—i.e., where student difficulties will require a concentrated effort—there is a wealth of information about what bait to use—principles of effective techniques, such as making predictions and real-time graphing.4
As good as the maps the literature provides are, they tell me nothing about
the dynamics of my students, where they are this class, what they have mastered
and what is still a challenge. Sonar works by sending out a pulse of sound
and listening what comes back from the object of interest. In the same way,
I constantly probe my students and listen to what comes back. Note that I
am not talking about tests here, final versions of a paper, or especially
final exams. Those are called summative assessments, and are more akin to
the report our fishing guide fills out for the authorities in charge of enforcing
fishing limits. They look backwards, summarizing what has happened during
a certain period of time, and are often a permanent record with official ramifications
(e.g. an ‘F’ on the final). Sonar, however, is a continuous probe
that serves as a guide for future action, rarely saved for future reference.
The fact that sonar systems have difficulty in distinguishing a mackerel from
a shark from a dolphin doesn’t really matter, since the next step is
to go fish there and see what happens. Formative assessment in the classroom
is a frequent—perhaps daily or even more frequently—probe of where
students are, that carries limited grade ramifications, and mainly serves
to guide instructor and students in how to proceed.
There are various techniques for formative assessment. The approach I mainly
use is Just-in-Time-Teaching,5 where students answer several open-ended questions
on the web before class. During the hour before class, I look over their responses,
selecting several representative answers, and use those student responses
as a touchstone to start the class. A small percentage of the overall grade
does rest on this, but the grading is based on effort, not correctness, so
students are free to take stabs at things they don’t fully understand,
or confess their confusion to me. Another approach popular with some of my
colleagues is using classroom student response systems.6 Several times during
the lecture the instructor will stop and pose a multiple-choice question about
the material just covered to probe how well the students actually understood.
Each student then uses a device similar to a TV remote to respond to the question,
the result gathered by a receiver and then plotted on the instructor’s
computer for the whole class to see. A low-tech approach is minute papers,
where students use the last few minutes of class to anonymously respond on
paper to a couple of questions, such as “What is the thing you found
most interesting, surprising, or though-provoking in this class?” and
“What is one thing that you are confused or unclear about?” On-line
discussion forums, self-quizzes and learning journals are some of the ways
that can be used for formative feedback, providing both us and our students
with some idea of where they are as the navigate the waves. Coupled with maps
that can be provided by educational research, annotated with our own experiences,
this can make our fishing trips more productive.
Of course, we don’t want just to catch our students; we want to communicate with them and the language we use to do that is often not the everyday language of our students. Problems occur when we fail to recognize how their language differs from ours, in order to adapt ourselves in some cases and help them adapt in others. In ordinary language, “acceleration” is often not clearly distinguished from velocity. After all, when you want to move at a high speed, you put your foot on the accelerator. “Acceleration” often also refers to speeding up, but not slowing down, since that is deceleration, or breaking. This is not just a matter of semantics. A student who has not come to recognize that acceleration and velocity are two different concepts will probably not recognize there is any conflict between the common preconception of force as proportional to velocity and Newton’s fundamental principle of force as proportional to acceleration. The student can also manage to use the right term but mean something different by it, masking a lack of understanding to the instructor and himself. The first step in addressing this problem is to become aware of how our language differs from our students’. Knowing that, sometimes we avoid using certain words with a gap if they aren’t really important, and other times make sure the students develop or receive clear definitions as to what a particular word means in this course, making sure they understand the definitions are important (e.g., it will be on the test). Another principle I often follow is to develop the concept before the definition to avoid having the terms interfere with developing understanding. For example, my students work through a lab where they use the computer to make graphs of velocity and acceleration and answer questions about how the graphs are related before I give a formal definition of acceleration.
Our students are not from a different planet, but the intellectual medium in which they move and work differs in important ways from ours. There is a difference in expectations of the learning process. There are often subtle but important differences in the foundational knowledge they bring to the classroom. Finally, there are differences in how language is used and understood. No, it is not a magic bullet that will solve all the problems. Mackerels are still mackerels and clams are still clams. However, the more we understand the differences in media, the better we can communicate subject and expectations. The more we understand the intellectual landscape below the surface and where they are, the better we can fish. The better to teach the sea.
Significant Results:
Ten Items for the Bottom Line
by Ken Kuehn, Distinguished Professor, Geography and Geology
As I write this piece, I am also detailing the syllabus for the first-time offering of our new senior capstone course in Geology. I am reflecting mainly upon the important discipline knowledge and skills our graduates need to succeed in society as professional geologists but I am also thinking about the overarching abilities, personal values, and attitudes that cross discipline boundaries and contribute to success, quality and enjoyment of life.
As educators, we must articulate several layers of desirable learning outcomes for our students and be able to assess them effectively. In my Académie Idéal, all instructors of the same course decide on a common set of learning objectives and assessments for that course, some of which address higher programmatic goals directly. The specific classroom activities designed to help students achieve those objectives are left to the creativity and talents of each individual instructor. Likewise, the larger programmatic goals are decided by a departmental committee-of-the-whole with some of those goals clearly embodying the higher values and vision of the university. Thus, every newly minted graduate from this academy begins life fully branded and well steeped. In actuality, however, we tend to provide an undergraduate experience that is far from seamless or consistent in terms of its outcomes; it is compartmentalized course-by-course and the results are not necessarily integrated or subsequently incorporated by our students into their everyday thinking and doing. Though we realize that learning is an ongoing process - not limited to a certain set of courses or the confines of a classroom - we tended to assume it all somehow comes together, that a little bit of polymathic magic occurs where every graduate solves the Rubik’s Cube of higher education to create a personal, coherent, bigger picture. But that just didn’t happen often enough and gradually a different view of our pedagogy emerged. Now we emphasize learner-centered education, active learning, engagement, social consciousness, citizenship, and so on, as both desirable and measurable results of the undergraduate experience.
So, what is it that we really want for, and from, our new graduates? What sorts of things should we imbue and assess before we send them out into the world with a WKU red towel? Here, in no particular order, is my list of the “Top 10 Significant Results,” a conglomeration of ideas and ideals from WKU documents (undergraduate catalog, QEP, etc.), similar documents from other universities, and personal thoughts.
Every new diplomate of a WKU baccalaureate program shall:
1. possess the ability for clear and grammatically correct writing. Can our graduate write a decent generic business letter or a technical report in the discipline?
2. possess effective oral communication and argumentation skills. Can our graduate give a persuasive presentation and recognize rhetorical propaganda?
3. have an awareness/appreciation for different modes of thought and methods of inquiry, especially the Scientific Method. Can our graduate recognize a testable idea; can they spot ‘pseudoscience’?
4. be able to effectively utilize information resources and computers. Can our graduate navigate the library, query an online database and use common software packages?
5. have the capacity to analyze new information and ideas from a disciplinary perspective and apply that perspective to a current issue or problem.
6. be able to discuss and reflect on contemporary issues in the community, nation, and around the world in the context of stewardship, sustainability, and socially responsible action.
7. respect the vast diversity of people and cultures around the world. Has our graduate experienced a culture other than his own?
8. appreciate the historical, political, and philosophical underpinnings of our nation and its democratic principles.
9. be able to appreciate and interpret a work of literature, poetry, or art.
10. understand the factors that contribute to one’s health and wellness, and frame a meaningful approach to a lifetime of personal growth and learning.
Do you agree with these? What would you add or subtract from the list? Can we/should we try to assess all these items? Let’s work on it together!
Top Five Ways to Bring About Significant
Results from Your Instruction
By Sally Kuhlenschmidt, Director Faculty Center for Excellence in
Teaching, WKU, Fall 2005
1. Start from clear student learning objectives
2. Develop learning activities that are intended to achieve the learning objectives
3. Use multiple methods of assessment to get a richer measure.
4. Compare your assessment items to those learning objectives-- do they align?
5. Use formative assessment as you teach, for example, use 1 - minute papers
[Give students 1 minute to write down the most important thing they learned
today. Collect them without names attached. Give some feedback the next class
time.]
Assessment
of Technology-Based Learning and Its Benefits for Adult Learning
by Jim Berger, Assistant Professor, Special Instructional
Programs
Technology and computers are becoming more prevalent in today’s workplace, higher education settings, and at home. Research has shown some benefits in their use with adult students, which can help improve students’ participation and engagement with the learning process for the following are reasons:
However, while these benefits sound good and helpful, they do not always bear out in the research. In an examination of adult learners using computer-based learning, Brown (2001) found that results were mixed. Brown examined 78 adults as they learned materials using computer-based training. Variables including self-efficacy, goal performance, goal mastery, age, education, computer experience, practice level and time on task were assessed. The results indicate that those who skipped around and picked and chose which activities to complete did not fare as well as those who completed the required exercises. What the research says about this is that many students who are given control over their learning through computer-based education tend to terminate the activity before reaching mastery level (Steinberg, 1989; Tennyson, 1980). We face a similar problem when working with adult students online or face-to-face. I believe one solution is to provide rewards for completing computer-based education so that students gain the necessary experiences and learn the materials they need.
In another study using computer-based online learning (Askov and Simpson, 2001), students were asked how they rated themselves on the knowledge they were asked to learn. Most, if not all, of the students reported that their knowledge greatly improved through the use of the online medium but what made the most difference was the role of the professor and the support provided, followed by the interaction among students. More research was suggested, indicating that much still needs to be determined before we can conclude that computer-based learning will improve significant results. What can be demonstrated is that computers are a tool which can be used beneficially for educational purposes. If we can engage students in the learning process and motivate them to proceed through all of the material and we, as instructors, provide the support and contact students need, then the likelihood of achieving a significant result is greatly increased.
References
With assessment of student learning, faculty often discover that students aren’t learning what they set out to teach. Traditional testing methods measure short-term retention, but what is learned may be quickly forgotten after the final exam. Newer, more innovative assessment methods seek to measure deeper level learning, i.e., increased ability for critical or analytic thinking, “big picture” thinking as well as details, and lifelong or job skills development as a result of class. If you believe the research-based thesis that assessment drives student learning, then you will want to closely consider whether your assessment practices effectively link student learning with the outcomes you seek.
The Internet hosts a wealth of excellent student assessment sites, including the following
The Faculty Center for Excellence in Teaching (FaCET) Resource Center is another good place to find resources on enhancing learning and assessing outcomes. Some available resources include:
About Us | Support FaCET | WKU
If you find it to be inaccessible, please contact Webmaster.
E-Mail facet@wku.edu -- Phone (270) 745-6508 -- Fax (270) 745-6145.
Write to the Faculty Center for Excellence in Teaching, 1 Big Red Way, Bowling Green, KY 42101-3576
Last Modified August 31, 2006. All Contents Copyright © 2000, Site created July 1996 Western Kentucky University