Undergraduate Academic Program Review

Frequently Asked Questions:

Why do you use such convoluted educational jargon?

One of the challenging things about participating in cross-disciplinary conversations is that each discipline has its own terminology to bring to the table. Thus, many of us have different words to indicate concepts of similar meaning, or we have one word or concept that has various meanings-you can see the difficulty when we try to compare practices, reach common understandings, or argue for different approaches. (A common language document will be created in the future).

A common language gives us the opportunity to speak across disciplines and still understand each other. And for most of us, that compromise meant setting aside our previous language and trading it in for a new language without losing any original meaning or integrity. To those not involved with this process of agreeing on a language, you may feel the language is "convoluted." We encourage your patience and point to your own disciplinary experience, from which you know that shared terminology is critical to a community of scholars.

What are some ways to do evaluation? Do they always have to be numerical? Are they always 'scientific'?

There are many acceptable methods of evaluation. Evaluation methods should be appropriate for your outcomes and consistent with your discipline's standards of assessment. If your discipline values and regularly uses numerical evaluation methods, those might be appropriate methods to adopt, especially if your outcomes are best measured in this way. It might be the case, though, that your outcomes are best measured with other forms of evaluation, such as portfolio reviews of student work, industry and/or alumni surveys, student capstone exams or presentations, and senior projects. These methods do not always have to be numerical, and they will be as scientific as your discipline's standards suggest. As you consider the evaluation methods, consider your outcomes and the disciplinary norms--design methods that are consistent and appropriate for both.

What is a rubric?

"The rubric is one authentic assessment tool, which is designed to simulate real life activity where students are engaged in solving real-life problems. It is a formative type of assessment because it becomes an ongoing part of the whole teaching and learning process. Students themselves are involved in the assessment process through both peer and self-assessment. As students become familiar with rubrics, they can assist in the rubric design process. This involvement empowers the students and as a result, their learning becomes more focused and self-directed. Authentic assessment, therefore, blurs the lines between teaching, learning, and assessment.

The advantages of using rubrics in assessment are that they:

• allow assessment to be more objective and consistent
• focus the teacher to clarify his/her criteria in specific terms
• clearly show the student how their work will be evaluated and what is expected
• promote student awareness of about the criteria to use in assessing peer performance
• provide useful feedback regarding the effectiveness of the instruction
• provide benchmarks against which to measure and document progress"

What is authentic assessment?

"Assessment is authentic when we directly examine student performance on worthy intellectual tasks. Traditional assessment, by contract, relies on indirect or proxy 'items'--efficient, simplistic substitutes from which we think valid inferences can be made about the student's performance at those valued challenges.

Do we want to evaluate student problem-posing and problem-solving in mathematics? experimental research in science? speaking, listening, and facilitating a discussion? doing document-based historical inquiry? thoroughly revising a piece of imaginative writing until it "works" for the reader? Then let our assessment be built out of such exemplary intellectual challenges.

Further comparisons with traditional standardized tests will help to clarify what "authenticity" means when considering assessment design and use: *

• Authentic assessments require students to be effective performers with acquired knowledge. Traditional tests tend to reveal only whether the student can recognize, recall or "plug in" what was learned out of context. This may be as problematic as inferring driving or teaching ability from written tests alone. (Note, therefore, that the debate is not "either-or": there may well be virtue in an array of local and state assessment instruments as befits the purpose of the measurement.)
• Authentic assessments present the student with the full array of tasks that mirror the priorities and challenges found in the best instructional activities: conducting research; writing, revising and discussing papers; providing an engaging oral analysis of a recent political event; collaborating with others on a debate, etc. Conventional tests are usually limited to paper-and-pencil, one-answer questions.
• Authentic assessments attend to whether the student can craft polished, thorough and justifiable answers, performances or products. Conventional tests typically only ask the student to select or write correct responses--irrespective of reasons. (There is rarely an adequate opportunity to plan, revise and substantiate responses on typical tests, even when there are open-ended questions). As a result,
• Authentic assessment achieves validity and reliability by emphasizing and standardizing the appropriate criteria for scoring such (varied) products; traditional testing standardizes objective "items" and, hence, the (one) right answer for each.
• "Test validity" should depend in part upon whether the test simulates real-world "tests" of ability. Validity on most multiple-choice tests is determined merely by matching items to the curriculum content (or through sophisticated correlations with other test results).
• Authentic tasks involve "ill-structured" challenges and roles that help students rehearse for the complex ambiguities of the "game" of adult and professional life. Traditional tests are more like drills, assessing static and too-often arbitrarily discrete or simplistic elements of those activities."

What are formative and summative assessment?

"Formative" and "summative" refer to the purposes of assessment activity and how assessment information is used.

Formative assessment is a feedback process that provides information you can use to fine-tune or modify what you've been doing. For example, the written comments you make on a student's essay are formative feedback to the student. Evaluating your students' senior project reports to see if your program should include more training in professional writing provides formative feedback to your curriculum committee.

Summative assessment is a judgement about how well you're doing, usually made by comparing your performance to some standard or to the performance of others. The grade you give the student's paper is a summative assessment. Asking your graduates' employers how your students compare to graduates from other schools can be a summative assessment. Sometimes the same information serves both purposes, as when you use separate grades on different parts of a student's work to point out strengths and weaknesses. Similarly, asking how the general picture of your seniors' work that you get from a year's senior projects matches the standards your program has set for each of your learning objectives gives both a summative comparison against your standards and formative information about the program's strengths and weaknesses.

Both roles will be used at NKU for assessment. The information you gather is first for your own purposes, to monitor and fine-tune your program, and one of the main questions in program review is whether you are paying attention and using to that information. Some of the things you learn will also be useful to help tell your story to administrators, accreditors, peers inside and outside NKU, and your other constituents.

What is a student portfolio?

"Portfolios are scarcely a new concept, but renewed interest, fueled by the portfolio's perceived promise for both improving assessment and motivating and involving students in their own learning, has recently increased their visibility and use. The definition of a portfolio varies some, but there seems to be a general consensus that a portfolio is a purposeful collection of student work that tells the story of student achievement or growth. (Portfolios are not folders of all the work a student does.) Within this limited definition there are portfolio systems that: promote student self-assessment and control of learning; support student-led parent conferences; select students into special programs; certify student competence; grant alternative credit; demonstrate to employers certain skills and abilities; build student self-confidence; and evaluate curriculum and instruction.

Because there is no single correct way to "do" portfolios, and because they appear to be used for so many things, developing a portfolio system can spell confusion and stress, much coming from not realizing that portfolios are a means to an end and not an end in themselves. More specifically, confusion occurs to the extent there is lack of clarity on: (a) the purpose to be served by the portfolio, and (b) the specific skills to be developed or assessed by the portfolio.

It is important to keep in mind that there are really only two basic reasons for doing portfolios--assessment or instruction. Assessment uses relate to keeping track of what students know and can do. Instructional uses relate to promoting learning--students learn something from assembling the portfolio."

What is the difference between assessing a program and assessing a student?

Faculty members ask if they could just use student grades as data for assessing academic programs stating that if a program outcome is related to a particular course or assignment, the grades for that course or assignment should indicate the degree to which students are able to meet the outcome. That may be true in certain rare cases, but on the whole, student grades don't provide the best data for program assessment because the two kinds of assessment are different in important ways.

First, they have different purposes. In the classroom, student assignments are designed to with the goal of helping students achieve the learning objectives specific to the class and the assignment. The assessment is used as a teaching tool, for guiding and testing student learning. Program assessment's purpose is to provide faculty with the information they need to improve their programs, to determine the degree to which the program is enabling students to meet program outcomes and to propose changes in the program as indicated.

Another difference is that student learning outcomes in individual classrooms are likely to be different from program outcomes. Learning outcomes are specific to a class and to the needs of students in that class and may change from teacher to teacher and semester to semester. They provide a framework for student learning. Program outcomes tend to be broader and more general, focused on what a program's courses have in common rather than the individual outcomes of each course. And because program assessment takes a broader view, there could be program outcomes that do not appear as specific learning outcomes in any class. This difference between outcomes means that the criteria by which a teacher assesses students and by which faculty assess a program are also likely to be different.

In addition to these differences, there is also the problem that grading processes vary across faculty members, across a single faculty member's courses, across semesters, and even across particular assignments in a single course. This natural variability makes it difficult to use grades as a reliable indicator of student abilities and thus as data for assessing a program. However, if a program gives all its majors a highly reliable and valid test or other assignment directly related to a program outcome, grades on that assignment could be used for program assessment.

The fact that student grades are typically inadequate as data for program assessment doesn't mean that student work itself--research projects, essay exams, lab reports, literature reviews--is also inadequate as data. Indeed, student work may be the very best data for many program outcomes. The difference is that faculty who are doing program assessment are most likely going to take a perspective on that work that differs substantially from the teacher of the course, applying different criteria for different purposes. Thus the same data may be used for both student and program assessment, but the way those data are used is not the same.

Assessing students and programs are quite different activities with quite different purposes. However, they do have one critical goal in common: student learning. Student assessment is a teaching tool designed to encourage and evaluate student learning. Program assessment is an institutional tool designed to enhance academic programs in order to improve student learning. They each play an important role in creating a better learning environment for our students.

We assess individual students in every course and give them grades. Why aren't course grades sufficient as program assessment?

See above question "What is the difference between assessing a program and assessing a student?" in the program review process. Also see

• the answers to "How is 'assessment' different from regular evaluation of students?" and "Why can't regular grades be used for assessment?" in the FAQ from the University of Texas System.
• the University of Massachusetts- Amherst's booklet "Program-based Review and Assessment"
• the section "Why aren't course grades enough?" in California State University-San Bernadino's Fall 1998 Outcomes Assessment Forum newsletter.

Grades can be used for program assessment if they relate to specific program outcomes and grading methods are consistent across faculty and courses.

• For example: "The local measure of 'grades,' by themselves, do not necessarily ensure that a student has achieved a particular outcome. However, grades can function as appropriate measures when faculty have taken steps to ensure that a particular course assignment clearly corresponds with an intended objective. More specifically, grades on tests or papers (as opposed to course grades or overall GPA) can be used if they are designed to assess a particular competency in a program."(Program Guide for Outcomes Assessment, Geneva College, p. 11).
• The University of Texas's FAQ recommends Walvoord & Johnson's Effective Grading for information about this approach to grading.
• This is quite different from compiling course grades from selected courses or across the program, or counting the number of courses in which students received passing grades. A program assessment process based on this kind of grading requires a lot of work and coordination across courses.

Why is this program review necessary? We already do student evaluations and peer reviews, and we get a great deal of feedback on the success of student learning from our essay exams and term papers, not to mention student questions and responses in our classes, which are small enough to permit one-to-one interaction.

Although student evaluations provide useful information to individual instructors, they are not a direct or reliable measure of learning outcomes, focusing mostly on students' perceptions of their experiences in a class. Essay exams, term papers, and grades in a specific course reflect achievement by the standards of one teacher, and there is plentiful evidence that teachers bring to bear many different standards and beliefs when they look at student work. The best way to judge whether an entire program is achieving outcomes created for that program is to conduct separate evaluations of student progress across the department's curriculum and within certain key courses. There is also great value in having teachers work together to formulate expectations and talk about what it means for student to meet those expectations. In fact, the usefulness of evaluating specific course assignments can be instrumental in these discussions for program improvement. Finally, this kind of assessment is formative-it's designed to help departments to do better at what they have articulated as their primary goals for student learning.

What about skills and knowledge that don't lend themselves to measurement? Do they not really count anymore? Are we now supposed to teach only what is measurable?

"What does one mean by 'measure'?" Webster's New Collegiate Dictionary defines the verb measure as: "to estimate or appraise by criterion." One does not have to count or summarize by a number in order to make that appraisal. One does have to establish an objective, understand the outcomes that will give evidence of its achievement, observe, record and appraise that student's behavior in order to perceive and present the evidence of achievement, and learn from the evidence either that students are heading in the directions they need to, or that someone needs to modify the course. One does have to discern that certain student works are members of one set, and others are members of another. One does have to establish the criteria that will allow the discernment. All of this can be wonderfully achievable without numbers. Sometimes it can be wonderfully achieved with numbers. But in no way is assessment necessarily dependent on numbers. Everything that we teach is capable of appraisal by criterion.