Assessment
Tools and Terms
Understanding the Terminology
A
Glossary of Measurement Terms, ERIC Digest
Harcourt Glossary
of Measurement Terms
Quantitative Methods
Quantitative methods of assessment
require the use of standard measure (surveys, questionnaires, etc.)
sot that the varying perspectives and experiences of people can be
fit into a limited number of predetermined response categories to which
numbers are assigned. This way it is possible to measure the
reactions of many people by way of a limited set of questions. Examples
of quantitative methods include satisfaction surveys, and needs surveys.
Qualitative Methods
Qualitative methods of assessment
include detailed descriptions of situations, events, people, interactions,
and observed behaviors; use of direct quotations from people about
their experiences, attitudes, beliefs, and thoughts; and analysis of
excerpts or entire passages from documents, correspondence, records,
and case histories. Qualitative assessment is conducted in a
natural setting, where people work, play, or live. It is descriptive
in nature, and is not concerned with using standardized survey instruments. Examples
of qualitative methods include focus groups and interviews.
Questionnaires
Item Construction: Question
Formats
1. Yes/no
questions: Also can be agree/disagree, satisfied/unsatisfied, etc. Generally
questions with only two possible answers
2. Multiple
choice: Questions followed by a set of fixed response choices. Can
also include “Other” as a choice.
3. Rankings: Respondent
numbers from high to low a list of forced choices. Ranking
criteria is usually importance.
4. Likert
scales: Questions employing a five-point response scale with
categories such as strongly agree; agree; undecided or neutral; disagree;
and strongly disagree (or some variation of this 5-point example). Used
to assess attitudes and opinions.
5. Other
rating scales: Items that use a variety of forced choice response
categories to evaluate such things as quality (excellent, good, fair,
poor/ better, the same, worse), behavior frequency (never, sometimes,
frequently) and importance (not at all, somewhat, very).
6. Open-ended
questions: Free response. Question followed by a blank
space for subject to put his or her answer.
Item Construction: Characteristics
of Good Questions
1. Not
too specific, not too general (better to err on the specific side)
2. Direct/obvious
intent (not vague)
3. Respondent
willing to provide answer (not too personal)
4. Use
the respondent's language
5. Minimize
social desirable responses
6. Value-neutral
(not leading)
7. Short
Telephone Surveys
The first step is to determine
if a telephone survey is appropriate for your assessment needs. Usually
telephone surveys are used for short, to-the-point assessment questions.
Often, a paper survey, focus group, or one-on-one interview with
open-ended questions or more specific questions is used as a follow-up
to a telephone survey to gain more information or clarify telephone
survey results.
Some of the advantages of
using telephone surveys include:
1. Rapid data collection
2. Assurance that questions
are clear to the respondent
3. Assurance that instructions
are followed
4. Relative anonymity
of the respondent.
Some disadvantages include:
1. Inability to access
certain telephone numbers
2. Limited complexity
of questions
3. Limited open-ended
questions
For an example of the use
of telephone surveys, look at Penn State Pulse, http://www.sa.psu.edu/sara/pulse.shtml a
telephone data collection system at Pennsylvania State University.
Individual Interviews
Interview Types
1. Structured & Semi-Structured
· Verbal
questionnaire
· Formal
· Elicit
specific answers
· Build
on existing research and information
· Example: Suppose
you wanted to compare and contrast methods of evaluating student
performance among faculty in the hard sciences, social sciences,
and humanities. You could ask a fairly structured series
of questions about exams, writing assignments, homework, collaborative
projects, etc., and compare faculty's responses to the same questions
across disciplinary groups.
2. Informal
Interview
· Less
formal
· No
specific type of sequence of questions
· Exploratory
· Example: Interview
to determine student likes/dislikes about the student center. “What
do you like about the student center?” “What improvements
could we make in the student center?”
3. Retrospective
Interviews
· Can
be structured, semi-structured, or formal
· Respondent
needs to recall and reconstruct from memory
· Good
assessment of feelings, perceptions, less accurate in other regards
· Example: Telephone
interviews with non-returning students to determine why they left
the university
Types of Interview Questions:
1. Demographic
Examples: age,
gender, educational level, etc.
2. Opinion
or Values “What do you think” questions
Example: “Would
more frequent weekend activities motivate you to stay on campus
during the weekend?”
3. Report
of behavior
Example: “How
often do you visit the Preston Center?”
4. Feelings – Report
of emotions or inner state
Example: “Do
you feel like you have a close group of friends on campus?”
Focus Groups
Overview
· A
qualitative research technique based on a group interview of 8-12
people
· Conducted
by a trained interviewer often using an assistant who takes notes
or performs other support functions
· Uses
an interview guide or protocol (list of questions) to direct the
interview
· Results
are directional rather than definitive
General Uses
· Stand-alone
assessment
· Complement
to questionnaire surveys and other evaluation methods – pre-cursor
to questionnaire survey design OR post-hoc analysis: meaning
of research results to target population
Examples of Using Focus
Groups
· Assessing
entering student needs and perceptions
· Evaluating
student support program quality, availability, and effectiveness
· Assessment
of needs
Drawbacks
· Provides
limited image of the topic in question
· Not
representative or normative
· Group
dynamics can suppress or otherwise alter expression of individual
viewpoints
|
