Programmatic Evaluation & Educational Assessment Plan

Preamble

Evaluating and improving the School of Pharmacy’s performance is essential for the school’s continuing success. Programmatic evaluation of the Doctor of Pharmacy program analyzes the extent to which the school achieves its overall mission and goals, including research and other scholarly activities, service and practice. Assessment of the Doctor of Pharmacy program focuses on student learning and curricular effectiveness. Both are undertaken as part of systematic improvement efforts. This document spells out the principles that guide each of these processes, outlines how these processes connect to other activities and groups within the school, and identifies who holds responsibility for deciding upon and for carrying out specific activities.

I. Overview of the School of Pharmacy (SOP)

1. Mission
   The mission of the University of Wisconsin–Madison School of Pharmacy is to educate, train, and provide life-long learning opportunities for students, pharmacists and scientists, while creating, disseminating and applying new knowledge based on research in the biomedical, pharmaceutical, social and clinical sciences to enhance the quality of life through improved health.
2. Administrative Organization
   The school’s internal organization supports activity in all mission-related areas. The faculty is organized into four academic divisions: Division of Pharmacy Professional Development (DPPD), Pharmaceutical Sciences (PS), Pharmacy Practice (PPD), and Social/Administrative Sciences (SAS). School-wide efforts are led by numerous standing and ad hoc committees. Administrative offices provide support to all functional areas throughout the school.
3. Academic Programs
   The school offers five academic areas of degree programs:
   1. Doctor of Pharmacy (PharmD) – a professional doctoral program
   2. BS in Pharmacology/Toxicology – an undergraduate biomedical science program housed within the Pharmaceutical Sciences Division
   3. MS in Healthsystem Pharmacy – offered jointly with a 2-year Administrative Pharmacy Residency and overseen by faculty in the Pharmacy Practice Division
   4. MS and PhD in Pharmaceutical Sciences – overseen by faculty in the Pharmaceutical Sciences Division
   5. MS and PhD in Social and Administrative Sciences in Pharmacy – overseen by faculty in the Social and Administrative Sciences Division
4. Evaluation and Assessment Responsibilities
   Programmatic evaluation and educational assessment activities involve all members of the school’s faculty and staff. Lead responsibility for these activities reside with several specific groups. The Assessment Committee, a standing committee in the school whose members are appointed by the Dean, is charged by the Dean to plan, oversee and review all evaluation and assessment efforts. The Office of Student and Academic Affairs, which includes the Associate Dean for Academic Affairs and the Director of Assessment, provide ongoing administrative support for evaluation and assessment activities. The Academic Planning Council, a body elected by the faculty and consisting of representatives of the school’s academic divisions, provides oversight through regular review of evaluation and assessment policies and planning.
   
   The Dean, assisted by the associate and assistant deans and the division chairs, is responsible for assuring the quality of all functional areas within the school. Evaluation and assessment efforts engage numerous school committees and administrative units. These groups and their evaluation/assessment-related roles are illustrated in Figure 1 below.
   
   

II. Programmatic Evaluation

In 2015, the Provost’s Office required all programs to submit learning goals for each degree program. In 2016, each program supplied the Provost’s Office with an assessment plan to assess the stated learning goals. The first annual report for each of these programs was due in 2017. Given that all programs are now being assessed per individual assessment reports to the Provost’s Office, this document narrows its focus and become the assessment plan for the Doctor of Pharmacy (PharmD) program.

1. 1. Principles and Approach
      Successful programmatic evaluation is an ongoing, systematic effort that focuses on processes and outcomes that are important, employs processes that are minimally intrusive and effectively link into ongoing activities, applies metrics that are meaningful, and yields results that are useful in identifying areas and directions for improvement.
      
      This plan divides the evaluation into two areas of foci: the Strategic Plan and the PharmD program. While programmatic assessment typically focuses on the program and student learning outcomes, including the strategic plan here acknowledges that this program maintains a primary focus in the School of Pharmacy due to its size and complexity with accreditation requirements. Because of this complexity, it is appropriate to annually assess the placement and fit of PharmD program within the school’s current strategic initiatives.
   2. Strategic Planning
      The school’s mission drives and informs its Strategic Plan and the identification of strategic initiatives. Thus, linking programmatic evaluation with strategic planning assures that evaluation focuses on processes and outcomes that are important to the school, links to ongoing activities, and produces results that can be readily incorporated into improvement efforts.
      
      History of Plan Development
      The school’s strategic plan is a continually evolving document where initiatives are prioritized through changes in the internal and external environments tied to the school. The strategic planning process involves five steps. Plan development begins with review/revision of the school’s Mission, Vision and Guiding Principles statements and review of the University’s strategic plan. Next, trained facilitators assist the faculty, staff, students and invited stakeholders in the development of the over‐arching goals within the strategic plan along with the objectives that are intended to meet those goals. The Academic Planning Council (APC) is charged with crafting the wording with input from faculty, students, staff and other stakeholders. A faculty/staff retreat is used to review the strategic plan, offer feedback on the objectives, and formally approve the plan. Finally, each year the APC revisits the strategic initiatives, updating them as necessary, and identifying an annual action plan. The semi‐annual Board of Visitors meeting is a forum for the dissemination of the strategic priorities and affords external stakeholders an opportunity to comment. Students from all academic programs are invited to participate in the strategic planning process.
      
      Strategic Initiatives
      The APC, in consultation with faculty, staff, students and other stakeholders, revise strategic initiatives for the following academic year at the end of every spring. The school’s formal committees, as reporting “owners” of the strategic priorities, evaluate and establish new goals. These strategic initiatives include one or more actionable items.
      
      The strategic plan creates the framework for the school’s strategic initiatives and action plan; however, it is not the sole driver of decision‐making within the school. Governance processes allow for targeted outcomes that may not be specified in the strategic plan (such as curricular revision) creating a path for initiatives to be introduced, discussed, and approved by the faculty.
      
      Evaluation and Monitoring
      To support monitoring of the strategic plan, strategic initiatives identify related goals and outcomes, and key stakeholders (co‐owners) for each priority. Key stakeholders and co-owners are identified throughout the school’s committee structures allowing a wider audience to influence strategic priorities. Committees are strongly encouraged to address the assigned strategic priorities in their charges annually.
      
      Co‐owners (commonly associate and assistant deans) document activities and progress, and allow ongoing monitoring by faculty, students, staff, and other relevant groups through associate committees. Those assigned as “owners” for specific strategic priorities or action items are responsible for gathering and summarizing updated information regarding progress and achievement of action items. Progress on each strategic priority is reported annually by May 1 to the Academic Planning Council, the group in charge of the annual Strategic Plan progress summary report to the dean. More importantly, this summary will be incorporated to further develop the strategic initiatives and action plans for the following academic year. Progress reports for each initiative are also shared with topically‐related committees.
      
      Reporting
      Strategic Plan progress updates will be compiled into an Strategic Plan progress summary (to be later incorporated into the annual assessment report) written by the Director of Assessment. The report will be distributed to relevant bodies for review and follow‐up. These include:
      • The Dean, to provide guidance in constructing charges for SOP groups, including standing and ad hoc committees of the school, academic divisions, and other administrative units.
      • The Academic Planning Council, to assist them with: 1) reviewing School performance, 2) developing strategic initiatives and action plans, and 3) reviewing revision of this plan.
      • The Faculty and Academic Staff of the school, for review and for informational purposes.
      • The students and other appropriate stakeholder groups, for informational purposes.
   3. Program Assessment: Doctor of Pharmacy (PharmD) Program
      
      Guiding Principles
      Several general principles underlie the school’s approach to educational assessment.
      1. Programmatic assessment provides meaningful evidence about student learning outcomes to inform the school’s decisions that affect the educational environment. This evidence can include direct and indirect measures of student learning, graduation outcomes, and feedback on the learning experience. Evidence becomes more meaningful with a comparative context, such as changes over time or contrasts with peer institutions, and performance goals.
      2. Program assessment processes are sustainable and intended for long-term use. The initial scope and objectives of a project guide the complexity and resources necessary for the assessment process to ensure that it is manageable. Changes in the context or situation for a project are an opportunity to refine the assessment process.
      3. Program assessment is a collaborative process. The needs and expertise of faculty, staff, students, and external stakeholders must be considered when identifying assessment needs, developing measures, and interpreting results. Collaboration throughout the assessment process promotes the relevance, utility, and credibility of the data collected.
      4. Program assessment involves developing and collecting accurate, representative measures and metrics that convey student success and the school’s success. Metrics are guided by the school’s strategic plan, accreditation requirements, and accountability to stakeholders.
      5. Program assessment is a transparent process. Assessment results are shared fairly, openly, and sufficiently with audiences who need the results to inform their decisions. Assessment results are presented with a relevant context when applicable for appropriate perspective.
         
         Course Level Assessment: Instructional Evaluation and Tools
         Instructional course evaluations are completed every semester for all PharmD courses and experiential sites. Student feedback is collected and reviewed by the instructor, course coordinators, and division chair. Instructors continually revise and update course content and activities based on student feedback. Course level assessment feeds into the program assessment as a data source and as a point of revision. Significant course changes may result in a formal course change process to update the course with the university course guide.
         
         Overview
         Educating future pharmacy practitioners is accomplished via the school’s Doctor of Pharmacy (PharmD) professional degree program. This program began as an entry-level degree program in 1997 and graduates approximately 135 students per year. Program graduates must pass external licensing examinations in order to practice pharmacy.
         
         Roles and Responsibilities
         Assessing an academic program involves 4-step process. The Curriculum Committee holds primary responsibility for step 1 while the Assessment Committee holds primary responsibility for steps 2-4.
         1. Articulating the educational goals and objectives for the Doctor of Pharmacy program. At present, 15 educational outcomes are identified as capturing the knowledge, skills, attitudes and values required of all graduates of the UW–Madison Doctor of Pharmacy Program.
         2. Identifying and describing instruments, methods, and timing for assessing student learning outcomes (specified in step 1) and overall curricular effectiveness.
         3. Developing feedback mechanisms so that assessment results generated in step 2 are used for improvement of the Doctor of Pharmacy program.
         4. Establishing a process and timetable for continuous improvement in steps 1-3.
         
         As indicated above, the school’s Director of Assessment provides administrative leadership and the Assessment Committee and Curriculum Committee play a core role in this process compiling and reviewing assessment results, and developing the annual assessment summary. This report:
         1. Summarizes the extent to which each of the Educational Outcomes expected of Doctor of Pharmacy students are achieved,
         2. Identifies areas in which the Doctor of Pharmacy program is successful in achieving its educational mission, as established by the school,
         3. Suggests revision of the Doctor of Pharmacy curriculum or pedagogy in areas where assessment results detect problems or identify areas for improvement, and
         4. Suggests revision in the measures, methods and/or overall plan for educational assessment, if the committee feels that current practices do not adequately capture needed information.
         Educational Assessments
         The School of Pharmacy uses a variety of assessment instruments and methods. Direct assessments of student performance serve as the primary means for assessing student learning outcomes. These are complimented by indirect assessments that provide useful information regarding diverse aspects of program operations, student experiences, and student achievement.
         
         Student Learning Outcomes
         The Outcome Tracker is the system of embedded assessments that defines key assessments for each Educational Outcome of the PharmD program. Each key assessment is completed regularly at specified points in the PharmD program. The Director of Assessment is responsible for coordinating with relevant course instructors in establishing and maintaining key assessment processes. The Assessment Committee is responsible for oversight of the Outcome Tracker. By the time a student has completed six semesters of required didactic coursework, at least 120 activities have been reported in their personalized Outcome Tracker. These activities span the gamut of course assignments including experiential and service learning activities, performance‐based skill assessments, lab practicals, written exams, and assignments. Educational outcome performance reports are discussed at various school‐level committees.
         
         Curricular Effectiveness Program operations, student experiences, and student achievement are assessed using a wide variety of tools and techniques. The aggregate summaries provided through Outcome Tracker are used in conjunction with other assessments to help evaluate curricular effectiveness. Survey tools include: the AACP Curriculum Quality Surveys, Advanced Pharmacy Practice Experience (APPE) site evaluations, and instructional course evaluations. Qualitative information is gathered through exit interviews with graduating students and forums with clinical instructors. Records that are reviewed include Student Promotion case analysis, graduation and attrition rates, and grade point averages and trends.
   4. Summary Reporting
      As the summaries of all assessment tools are combined, a snapshot of programmatic achievement develops. This information is reviewed and recommendations are made for future improvements. The Director of Assessment, with assistance from the Assessment Committee, combine the Strategic Plan progress summary with Outcome Tracker and curricular effectiveness data to create the Annual Assessment Report. (See Figure 2 below for Reporting timeline).