If you do your assessment well and if you gather multiple sources of evidence, then you will worry less about getting others to accept your proof. However, keep in mind that since assessment is research, just like any research, there will always be those who criticize your methods and your rigor. Most importantly, if you use your findings to make your decisions, and consequently your program improves over time, your critics have little ground on which to argue.

Questions about General Education Requirements

What do I do about including general education requirements in program review process?

General Education courses have student outcome requirements defined. Programs should ask "What skills, abilities, and/or areas of knowledge are the department's general education course offerings for non-majors intended to produce?" and "whether general education offerings for non-majors produce the intended outcomes.".

Ask the question "What do we want every NKU graduate to be able to know or do as a result of the combination of their discipline specific and general education requirements?" Thus programs are encouraged to think of this question when constructing their undergraduate academic program review and address those aspects that are most meaningful to the program faculty.

When will NKU begin to review Minors and General Education courses?

Review of minors in your program can begin now. Programs can currently articulate outcomes for their minors and approach assessment of their minors through courses.

As for the formal review of general education courses, there is no formal process currently defined. During our reaffirmation process, faculty submitting general education courses were required to l create student outcomes for each course submitted. The general education curriculum committee reviewed and approved each course prior to approval.

Assessment of the general education program will be a project undertaken by the Office of Curriculum, Accreditation, and Assessment during 2005. Faculty and departments will be consulted as the process is defined.

Existing Accreditation Reviews

What do we do about our outside evaluations, external accrediting procedures, etc.? Can we just submit the reports we have for those constituents?

In the spirit of not reinventing the wheel, as well as for consistency, we encourage you to use information and reports you have devised for other audiences. In fact, that is one of the reasons we try to schedule the program review for the program one year after any accreditation visit-so you can use the information you supplied your accreditors and so you can respond to any feedback you received from them.

If materials are being produced for the University's periodic accreditation review by the Southern Association of Colleges and Schools (SACS), those materials may be used in the program review portfolio. In these cases, the program's portfolio must also include

1. the review teams' response to the materials if one is available
2. materials on any topics listed that are not included in the external review materials, and
3. an index, table of contents, or concordance that indicates where the material on each of the topics is located in the external review materials or elsewhere in the portfolio."

Where do we find the time for this?

Where do we get the time to do these reviews, and how can we give the involved faculty credit for their work?

Working smart means there will be a minimal amount of time required for the program review process. Much of the assessment activity may already take place as part of your compact planning process or as part of your external accreditation process. As we improve our teaching, we already review our course-based teaching and can build on the course-based efforts to improve our programs. It is also important to make effective use of the time spent. This can be accomplished by seeking guidance with your discipline and/or within the University, and by taking advantage of the additional resources in place for continuous assessment. The reporting process is simply taking a snapshot of the continuous process and, thus, does not require an extra-ordinary amount of time.

For those of us who have never done this or any other outcomes assessment at all, please give us a breakdown of the estimated time that it will take a professor who teaches 4 sections a year with 50 students in each and wants to do a good job. Suppose, for example, that the professor (A) spends 20 hours a week on research, (B) spends 30 hours on teaching, and (C) spends 10 hours on required university business. From where is the professor to take the time to do this continuous assessment-based program review time and the other outcomes assessments that have been requested?

Like anything that we're taking up for the first time (e.g., teaching with technology), continuous assessment will probably take additional time in the beginning. If done correctly, however, assessment should become a natural part of the individual instructor's and the department's teaching function. Because it is continuous, assessment need not be a large-scale effort absorbing vast amounts of resources in the form of time, staff, etc. Departments (and instructors) should be able to use the assignments that an instructor would design and assign to students anyway for its assessment purposes. Used properly, assessment will undoubtedly change teaching...for the better, resulting in enhanced teaching methods and more powerful student learning. And in the process assessment will supplant other planning activities associated with teaching as teaching strategies change, resulting in no net increase in time devoted to teaching. Once a system of really continuous assessment (i.e., small-scale, conducted as a natural part of the teaching function, in order to answer important questions about learning) is in place, it should not place unreasonable demands on faculty and staff.

What happens as a result of assessment
and program review?

What happens to my program review once I submit it?

According to the Program Review Guidelines, the report should first be reviewed and approved by the department and then submitted to the department's college for review. The college submits the report and the dean's comments to the Steering Committee who will review the document, summerize findings and forward it to the Provost. CUPR will read over your document and prepare a recommendation concerning the effectiveness of your assessment. The committee will use a program assessment rubric to determine how thorough and meaningful your assessment portfolio is, focusing particularly on the quality of questions and appropriateness of collected data to answer those questions, the usefulness of those answers in various forms of program planning and decision making, and evidence of an ongoing commitment to continuous monitoring and improvement. The committee's remarks and recommendations will be forwarded to the Provost.

Who is going to read and evaluate our program review document? And how intrusive are those readers/evaluators going to be?

The program review document will be submitted to the Provost's office. It will first go to Steering Committee. For the foreseeable future, a committee called the Steering Committee comprised of representatives from each of the colleges and a chair appointed by the Provost has been formed to lead the program review process.

This is not a checklist. The Steering Committee does not expect every program review to be conducted or reported in the same manner. In the end, the committee will judge whether a group of faculty has made a strong effort to

1. state what it is that they are doing (with regard to undergraduate education),
2. measure to what extent they are accomplishing their goals,
3. provide data to support how they know this (#2) to be true,
4. show who has analyzed the data and how the information has been used to improve the program, and
5. discuss any changes made to the assessment process, as well as the curriculum, in order to ensure that the process works effectively in helping the faculty to improve the program.

The Steering Committee's job is not to decide what a program's objectives, outcomes, and assessment process should be, but to ask whether they meet general criteria for effective outcomes assessment. For example, program review reviewers will ask whether a program's faculty have found ways to actually measure their stated outcomes. If not, they may recommend that the faculty think about changing the outcomes to make them more meaningful and thus more easily measured and suggest resources to help the faculty do this.

What is the bare minimum we have to do to meet University requirements?

A department must compile a program review report that includes all interdisciplinary and departmental undergraduate activities offered by the department at least once every five years. A summary of ongoing activities should be started at least one year before the program is to be reviewed. The material in the report should reflect continuous and ongoing planning, information gathering, self-review, and use of results, which are not processes that occur only once per review cycle. The topics to be included in the report can be found in the Guidelines for Academic Program Review, effective Summer, 2004.

What do I get out of all this?

The faculty, departments, and the University get a great deal out of all this assuming we are really serious about student learning. Program objectives concerning problem solving, critical thinking, mastery of sophisticated concepts, and oral and written communication, for example, are very complex and take longer than a single course to develop. As a result, it's very important that instructors within a program have a common understanding of what students will know and be able to do when they complete the major. Equally important is instructors then designing learning experiences and assignments in their courses that intentionally develop such knowledge and abilities. More than before, we'll need to think developmentally about program objectives. For example, what should problem solving look like in a second-semester junior construction student, and how will I know it when I see it? Assessment will be an ongoing activity in the interests of student learning: the primary mechanism for determining whether we are bringing about the type of student learning we had hoped and then revising the learning environment accordingly if students are not learning in the way we had hoped. So achieving consensus on and then articulating good objectives that the entire program believes are important is the first step in coordinating our efforts effectively and efficiently in the interests of student learning.

Will this work make the process of learning about our students' capabilities simpler and more efficient?

I think that, in the long run, yes, the Program Review Process will simplify the process of learning how to improve our programs and showcase our accomplishments. It should also result in greater efficiency by gaining a consensus of faculty views, increasing agreement about ways to streamline programs, and enhancing the quality of student learning outcomes. Although it does take some time and effort to set the process in motion, the faculty will learn about good practices for supporting and evaluating student learning (such as the use of rubrics), and they gain more confidence about their own courses by comparing and contrasting what they do with that of the other faculty in their department. A little more work upfront that lays out our expectations for students' performances will assist students by making the evaluation of work more transparent to students and enable instructors to evaluate that work more easily with fewer problems and complaints later on. This kind of collaborative departmental work to reach consensus about our expectations benefits both the instructors and the students.

How will the University use the information gathered from the program review process the departments are providing? Are we going to be rewarded/punished for meeting/not meeting our objectives and outcomes?

Department heads, directors, deans, provosts, and presidents have always used information to fund or to deny funding to programs. That's how decisions are made; people fund what seems, to them at least, to be working or to be on the right track for success. The disconcerting part of that fact is that these decision-makers have not always had clear, relevant, or even accurate information on which to base those decisions. That truth was essential in driving the budget planning process, which allowed program and college deans to look at their discipline's current state, propose new directions and funding for those directions, and describe assessment efforts that would measure whether the new direction succeeded in successfully altering the current status. Thus, budget planning is different from Academic Program Review, which is tied to student learning and data-driven decisions about program improvement.

Because this process is about continuous improvement, you need some baseline data on which you can build your understandings and evidence of what is and is not working (program information gathered during this five-year cycle will be used as baseline data). In short, nobody expects any program to be "perfect" as is. We anticipate and encourage changes being made-changes that will reflect sensible and research-based adjustments to create the improvements you seek. Thus, those who actually "own" the program (the faculty and leadership in the department and college) will have real evidence that a program is or is not doing the task or creating the learning for which it was created. And if some element of the program is not working, program directors, faculty, and staff will then have useful information by which to consider what they might change about the program to correct or redirect that initiative.

Iit would be a grave mistake to see program review as a procedure for cutting funds. Rather, it would be far more accurate to see program review as an initial step for seeing what is and is not working and then for redirecting funding toward enhanced or new initiatives that research, data, and reason suggest would help improve any weak areas.

Who will determine and by what mechanism whether our review is sufficient?

The Steering Committee is the University committee that evaluates your program review, and that committee will forward its recommendations to the Provost's Office for final action. You will receive feedback on your program review along with any information on any areas of concern. However, it will ultimately be your department or program that should really be analyzing your review to see if it is sufficient. Did the review adequately address your objectives and outcomes and did it answer your questions regarding those items? Do you more clearly understand what your students do and don't know-what they can and can't do? And can you see places in the curriculum (and perhaps in other experiences as well) where your faculty can be more effective in ensuring student learning?

Because this process emphasizes continuous improvement, the review is meant to be a feedback and ongoing process, which means that there will generally be a next step or question to answer. The review that is sufficient for now will generally suggest additional steps for the next review.

By what mechanism, and to what degree, are these efforts going to be incorporated into the university's assessment of us?

One way to read this question is "How will university administrators use information from and about a program's assessment process when they make decisions that affect the college, the department, and the program?" Another is "How will program assessment and my participation in it affect administrative decisions about me, as an individual faculty member?" I will try to answer both questions.

With program review as an active process at NKU, university administrators will expect that your program's faculty are keeping track of how effective your programs and activities are and using that information in your program and curriculum planning. They will also expect you to use the information to support initiatives in your department's strategic plan and other requests for continued or expanded support and for changes in your program's situation. Program plans that affect undergraduate and/or graduate education won't automatically be approved because they are based on assessment data, of course, but plans that do take assessment data into account are more likely to be considered positively than plans that do not.

As an individual faculty member involved in undergraduate and/or graduate education at NKU, you are expected to contribute to your program's effectiveness and to take part in program planning. This expectation includes contributing to and paying attention to assessment and other program review information. In a real sense, this is part of teaching. Individual departments will differ in how they take account of this activity in assigning responsibilities and in evaluating faculty members, but as college and university administrators ask departments to actively use assessment and other program review information, departments will ask faculty to do the same.

This is not the most precise answer that some of you may want, saying exactly who will give how many points for what levels of performance on which program review measures. We can't be that specific. Your department and program decide what data you will gather and how you will use it. The information that's part of each planning request will be what's relevant to that request, and will vary with the program and the request. Appropriate roles for individual faculty members depend in part on their particular program's assessment process. The general answer, however, is that attention will be paid to programs' use of their assessment processes and to faculty members' participation in them.