1. What is a departmental assessment program?

A departmental assessment program evaluates the effectiveness of its undergraduate and graduate programs in terms of measurable student outcomes. The program consists of (a) lists of educational objectives for each of the department’s major programs expressed in terms of student learning outcomes; (b) measures of student achievement for each of the objectives; (c) methods of collecting data; (d) procedures for involving departmental faculty in reviewing and using the results of assessment, including revision of the assessment plan when necessary; and (e) annual collection, analysis, and reporting of the results of assessment.
 

2. What should be included in the department’s list of educational objectives?

The list of educational objectives for each academic program in the department should include knowledge, skills, and attitudes specific to the major plus, at a minimum, the three core University Studies objectives: locating and gathering information, written and oral expression, and critical thinking and reasoning. While the list of educational objectives should be comprehensive and detailed, it should still be feasible to assess every stated objective. Formulate each objective so that there is a credible connection between the objective and the method of assessing it.
 

3. Who should be assessed?

North Central Evaluators and Missouri CBHE staff emphasize that all majors should be assessed--typically as they near completion of their program. Thus, voluntary testing in which only some majors participate is no longer acceptable. Though some departments may wish to assess the learning of non-majors in their service courses and students transferring to other colleges and universities, the focus of assessment is generally on graduates from the department’s major programs.
 

4. How can the department insure that all students are assessed? What if students don’t want to participate?

The most viable solution is to integrate assessment into the curriculum. For example, a department might design internship evaluations so that they provide useful information about student performance on key objectives, or incorporate senior projects and exit exams into a capstone course, or pre-test students in an introductory course. Students will engage in assessment activities that are an integral, logical part of their education. In any case, since voluntary participation is highly unlikely to produce satisfactory levels of student involvement, both the benefits of participating and the costs of abstaining need to be made evident to the students in terms that make sense to them.
 

5. What if an appropriate nationally normed test of achievement in the major is not available?

Departments are not required to use nationally normed tests. In fact, North Central Evaluators discourage the use of nationally normed tests if they do not provide relevant information about student achievement in the major. One advantage of nationally normed tests is that they provide a comparative standard of performance; a disadvantage is that they often do not relate directly to a department’s program objectives. Popular alternatives to the nationally normed exam are locally developed exams (such as the exams used by our departments of Criminal Justice and Psychology) and performance-based assessments (a capstone project or a portfolio, such as the senior project in Geosciences or senior recital in Music). Locally developed exams are scored "objectively." Performance-based assessments typically use a criterion-referenced rating system.
 

6. Can you be more specific about acceptable and unacceptable measures of student learning?

North Central Evaluators (Lopez 13) distinguish direct from indirect measures of student learning. Direct measures include the capstone experience, portfolio assessment, standardized tests, certification and licensure exams, locally developed exams, essay exams blind scored by multiple scorers, juried review of student performances and projects, external evaluation of student performance in internships. Indirect measures include surveys, exit interviews, retention and transfer rates, length of time to degree, SAT and ACT scores, graduation rates, and placement and acceptance data. Grade point averages, grades in the major, faculty/student ratios, curriculum review documents, accreditation reports, demographic data, and other administrative data are not acceptable measures of student outcomes.
 

7. Is one good measure of student learning enough to satisfy the assessment requirement?

No. Departments are expected to use multiple measures of student learning. For example, a department might employ a capstone project, internship evaluations, writing test scores, ACT scores, California Critical Thinking Test scores (if University Studies decides to administer this test to all students annually), placement and acceptance data, exiting student interviews, and alumni survey data in its assessment program. One measure can be used for several objectives; a capstone project, for example, might be used to measure knowledge in the major, research skills, and communication skills. Departments are especially encouraged to use several measures for one objective; skill in writing, for example, might be measured by performance on the University’s writing test, performance on the program’s capstone project, and grades of the department’s majors compared with other students in English composition.
 

8. Whatever happened to value-added?

"Value added" is still with us. NCA Evaluators would be delighted to find that we are demonstrating value-added by testing both entering and exiting students. But, as Lopez writes, "pre-testing is not necessary if one is highly confident that students know little or none of the content they are to master through completing the degree program." (Lopez 12) Lopez stresses that pre-testing is particularly appropriate for transfer and graduate students, but the ongoing assessment of majors through various levels of development is always impressive. The assessment plans and activities of our departments of Music and Foreign Language are notable in this regard. The national model is Alverno College, where students are assessed continuously throughout their college careers. More fundamental than demonstrating growth, however, is the need to measure student achievement against clearly stated standards. Though it is quite difficult to demonstrate that a department’s programs are the primary contributor to student learning, it is less difficult to show that students are completing the department’s programs having reached an acceptable level of achievement relative to specific educational objectives.
 

9. Can you be more specific about standards? How do you set up and apply standards in assessment?

Standards constitute performance goals and should be defined in terms appropriate to the relevant method of measurement. Where comparative data are available, a department might define standards in terms of the percentage of students at or above a particular percentile. Missouri’s funding formula awards universities for the number of students scoring above the 50th percentile on nationally normed tests in the major. I have heard credible arguments that this is a stupid measure of success. An individual department, however, might have good reasons to state that all of its students should score above the 50th (or 65th, 70th, etc.) percentile on a standardized test in the major--provided that this is a meaningful expression of standards. Departments with licensure exams might want to state that no fewer than 95% of its students will pass the exam on the first attempt. And departments with a criterion-referenced capstone project (or internship evaluations based on specified criteria) might want to state that all students will receive at least a satisfactory score in each criterial area with 30% performing at a level higher than satisfactory.

Performance-based assessments present specific problems. Though standards are usually written into scoring criteria, performance-based assessments have little credibility unless results are analyzed by comparison to performance of students outside the department, by external review, or through conscientious discussion among faculty of the relative strengths and weaknesses of student performance. The Art Department’s evaluation of senior projects is strengthened by the fact that it employs an external judge, as is the evaluation of a department that identifies areas of weakness indicated by particular measures (for example, relatively weak understanding of the hypothetical method as indicated by performance in the senior portfolio) and proposes actions to strengthen them (for example, holding a department faculty workshop on teaching the hypothetical method). Whatever your approach, remember that statements such as "All graduating students passed the department’s exit exam" are not credible indicators of standards unless supplemented with appropriate analysis, interpretation and follow-up.
 

10. How can I add assessment to the already busy schedules of my faculty and students?

To the extent that one can incorporate assessment into daily practice, assessment will not appear as an additional burden. We need to find creative ways to incorporate assessment into curriculum and instruction so that it is part of our normal work load. The burden will seem unbearable to a chairperson who tries to pull together disparate elements of an uncoordinated assessment program on the weekend before the Departmental Annual Report is due. For the chairperson who plans ahead and fully involves faculty in the collection, interpretation, and use of assessment data, the burden will be less onerous.
 

11. When and where do I report the results of assessment?

The results of assessment are to be reported annually in the Departmental Annual Report, Section IX. The annual reports have been due late in April and are based on the preceding calendar year.
 

12. What should be included in the assessment report?

There should be entries in Section IX of your annual report, program by program, for each of the following:
 

13. Who will read my assessment report?

We assume that your assessment report will be shared with your faculty and reviewed by your dean. Ideally, your faculty or a committee of your faculty will have produced the report. In the end, all reports will be evaluated by the University Assessment Review Committee. The UARC evaluated all assessment components of departmental annual reports for the first time in summer 1997 and will continue to do so annually. In the future, selected results of assessment will be published in the University’s annual report to the Board.

14. If the UARC already has the department plan on file, why do I need to include it in the annual report?

Don’t include the whole assessment plan. In summary form, state each program’s educational objectives and identify the measures you used in the preceding year or plan to use in the future to assess them. This will help reviewers on the UARC to orient themselves to the remainder of your assessment report.
 

15. Are there required formats for reporting data?

Yes and no. You should receive WP003, ACT, MFAT, CCTT (contingent on University Studies assessment plan) and other data from Institutional Research in an MS Word table with a structure something like this:
 
 

This format shows the number of your graduates that have been assessed with the instrument and permits you to compare departmental programs, compare your department with the college, and compare your department and college with the University. If a test has subscores, separate tables for each subscore might be provided.

You may use a reporting format for departmentally collected data designed specifically for your assessment measure. If the measure is unique to the department, rows for college and university will not be necessary. Regardless of format, data on graduating seniors should be reported so as to support comparison across calendar years. For example, lists of students placed in jobs or in graduate schools within a year of graduation are useful in conveying the range and type of employment or post-graduate studies the majors are going into, but they do not give the reader information about placement rates and the success of the department in collecting information about placement rates. A table that does give such information might look something like this:
 
 

Suppose a department needs to report the results of assessment of senior capstone projects. One approach is to report to the number of graduates, number of graduates assessed, and number of graduates who passed. But if all graduating seniors must pass the senior capstone project, this would not be an especially informative table. More informative would be a table or set of tables that report results in terms of percentage of students achieving defined performance levels in specific criterial areas, as follows:

This style could be varied to report the result of internship and practica evaluations as follows:

Numerous styles of table are possible depending on the nature of the data to be reported. Remember that data should be collected so as to supply credible information about student achievement and to identify relative strengths and weaknesses. Tabular summaries of data should also support longitudinal comparison of departmental graduates and show how close the department is to assessing all of its graduates with that particular measure. Though there may be instances where tabular reporting is not appropriate, most data, including qualitative data, can be reported in tables if they are being analyzed within an explicit conceptual framework. Tables help the reader take in the results of assessment at a glance. However, tables should never be inserted into your report without comment. Analysis and interpretation are essential components of assessment, and the reader will want to know what you make of the data in the tables.
 

16. What do you mean by analysis and interpretation?

A number of things. Analysis helps the reader understand the data by describing general trends in the data and pointing out differences and similarities among data points. Interpretation relates data to the objectives they are supposed to measure, explores the relationships between multiple measures of an educational objective, qualifies, amplifies, draws inferences, and evaluates. Analysis and interpretation address questions such as the following: What do the data say about your students’ mastery of subject matter, of research skills, of writing and speaking, and so on? What do the data say about your students’ preparation for taking the next step in their careers? Are there respects in which your students are outstanding? Do they consistently score at the 85th percentile or above on certain subjects in the MFAT? Do they receive high praise from internship supervisors? Are they consistently weak in some respects? Are many of them getting good jobs, being accepted into good medical schools, reporting that they are satisfied with the education they have received from your department? Does their performance on senior capstone projects indicate that the literature research skills of your students are relatively weak? Are there areas where their performance is adequate but undistinguished--where you would like to see a higher level of performance? An attempt to address such questions through analysis and interpretation is an essential piece of any conscientious assessment program.
 

17. Do I have to use the results of assessment for the purpose of improvement?

In any given year, it may not be necessary or appropriate to launch a program improvement initiative based on assessment results. Still, NCA Evaluators have consistently faulted assessment programs on the grounds that the results of assessment are not being used to improve curriculum and instruction. The UARC hopes to see a significant increase in the number of departments using assessment for improvement, and if there are any efforts to improve departmental programs connected with assessment they should be reported. Reports of efforts to improve programs are telling indicators of a vital, ongoing assessment program. If your assessment program is not giving you useful information for program improvement, then this information (that the information is not useful) should be used to improve your assessment program. By a curious twist of logic, useless information thereby becomes useful.