Transcription
John Jay CollegeMaster of Public Administration (MPA)Assessment PlanDirectorsDr. Marilyn Rubin MPA-PPADr. Ned Benton MPA-IOMarch 2012
1. Outcomes Assessment Philosophya. Chronology of MPA Assessment ActivitiesAssessment of the John Jay College MPA program has been evolving for close to two decades,reflecting, to a large degree, accreditation standards imposed by the National Association ofSchools of Public Affairs and Administration (NASPAA), the national organization thataccredits (and reaccredits) MPA programs. Under NASPAA’s seven-year assessment cycle, allprograms seeking accreditation must undertake a self-study that includes an analysis of how theyassess program performance based on the standards in place at the time of the self-studypreparation.NASPAA’s original standards focused on program inputs, e.g., sufficiency in the number offaculty and courses offered. The second iteration of standards adopted in the 1990s added a layerof mission-based requirements to the inputs orientation. The third iteration, adopted in 2009,requires an additional focus on student learning outcomes. The John Jay MPA program will,however, be required to assess program performance as measured by student learning when itundertakes its next self-study for its reaccreditation in 2015.The assessment sections of the John Jay College MPA program self-study documents preparedfor NASPAA accreditation in 2000 and 2007 were consistent with the NASPAA programstandards. They show that we were formulating goals and objectives and related metrics toevaluate program success. However, reflecting NASPAA standards, our evaluation and metricsused to assess program efficacy did not place a particular emphasis on how well students werelearning the information the program provided.b. Current PhilosophyThe current MPA philosophy is to use the assessment process as a mechanism that will allow usto continuously improve student learning. We are defining what we want our students to learn,i.e. learning objectives, and using several metrics to assess how well they are learning. We arethus creating a feedback loop that begins with the identification of learning objectives and endswith the recalibration of our program based on student learning outcomes.Our goal is to develop an array of performance indicators that will help us to assess whetherstudents are mastering the material in the MPA programs and where they may need help. Ourassessment program is using several qualitative and quantitative metrics to identify students’strengths and weaknesses and to identify program areas that need improvement. Our learningobjectives have been developed with input from MPA faculty, all of whom are participating inthe assessment effort.2
c. Two MPA programsThere was one MPA program at John Jay College from the time of its inception in 1967 untilSeptember 2011. From 1967 to 1984, MPA students were offered a general program in publicadministration with no particular specialized focus. In 1984, the program initiated specializationrequirements so that students could develop knowledge and skills in particular areas such asHuman Resource Management and Criminal Justice Policy and Administration. One of thespecializations created was the Inspector General (IG) specialization which was given inconjunction with Baruch College.Starting in 1995, the IG specialization with Baruch was discontinued. It was replaced by the“MPA-IG track” with courses given only at John Jay College. The track became one of twooffered to John Jay students, the first being the “Regular” MPA track. The Regular track wouldcontinue to prepare students for administrative, staff and executive positions in public agenciesand non-profit organizations. The Inspector General Track would prepare student for positionsinvolving oversight of public agencies and independent organizations as well as privateorganizations regulated by, or contracting with, public agencies.Over the next 15 years, while the focus of both tracks remained on training for the public service,specific course requirements evolved to the point where more than 50 percent of the 42 creditsnecessary to obtain the MPA degree differed substantially. Due to this divergence, the New YorkState Education Department required that they be registered with different Higher EducationGeneral Information Survey (HEGIS) codes.1 Thus, starting September 2011, there is no longerone MPA program with two tracks. Instead, there are two MPA programs: (1) the MPA-PublicPolicy and Administration (MPA-PPA) Program and (2) the MPA Inspection and Oversight(MPA-IO) Program. The MPA-IO program is unique among MPA and Public Policy programsbecause it concentrates on the oversight, assessment and investigation of performance andintegrity in public and independent agencies, as well as private organizations regulated by orcontracting with public agencies.Since fall 2009, students have had the option of taking an on-line/hybrid MPA-IO Program. Theonline/hybrid program combines fully on-line classes with classes that meet face-to-face incoordination with conferences of the Association of Inspectors General, the leading nationalprofessional organization in this field.While the specific focus of the MPA-PPA and MPA-IO programs will differ, as did the focus ofthe two MPA tracks, they will continue to require several of the same core courses. Theassessment philosophy described above will also apply to both. Further, since the split into twoprograms did not take effect until September 2011, in the first three years of the MPA programassessment the great majority of students continue to be enrolled in the two MPA tracks thatconstitute the “original” MPA program.Accordingly, for the first three years of our five-year assessment cycle, the assessmentpopulation will be all students enrolled for the MPA degree. In the fourth year, we will begin to1The New York State Inventory of Registered Programs uses the HEGIS taxonomy to classify instructional programs in whichcode numbers represent subject areas.3
look at learning objectives specific to the two MPA programs – the MPA-PPA and the MPA-IO.It should also be noted that the learning objectives and assessment cycle described below applyto all students regardless of teaching modality, i.e. face-to-face or on-line.2. Learning ObjectivesCurrent NASPAA standards require all programs seeking accreditation or re-accreditation todevelop competencies in five “universal domains.” They are: (1) lead and manage in publicgovernance; (2) participate in and contribute to the policy process; (3) analyze, synthesize, thinkcritically, solve problems, and make decisions; (4) articulate and apply a public serviceperspective; and (5) communicate and interact productively with a diverse and changingworkforce and citizenry. Additional competencies are also required when programs have missionspecific objectives that all students must achieve, and for specializations. The learningobjectives for the John Jay College MPA programs have been formulated within the context ofthe five NASPAA domains as follows:Learning Goal 1: Lead and Manage in Public GovernanceLearning Objectives1.1 Acquire the knowledge, skills and competencies of leadership and management in publicgovernance and administration, including the organization, management, assessment andoversight of human, financial and technical resources.1.2 Understand the core functions, issues, and theories of human resources management.1.3 Understand the major theories of organizational structure and organizational behavior and thepreconditions necessary to organize effectively.1.4 Understand how governments finance projects and programs.Learning Goal 2: Participate in and Contribute to the Policy ProcessLearning Objectives2.1 Apply different policy process models to various management functions including IT,budgeting, HR, and strategic planning.2.2 Understand the influence of politics on policy choices and the practices of policy analysis.2.3 Understand the influence of economics on policy choices and the practices of policy analysis.2.4 Plan a research or policy analysis project that makes appropriate use of empirical researchtools such as data collection, sampling and statistical analysis.4
Learning Goal 3: Analyze, Synthesize, Think Critically, Solve Problems and MakeDecisionsLearning Objectives3.1 Demonstrate reading, writing and analytical skills necessary for decision-making.3.2 Define and diagnose decision situations, collect and analyze data, develop and implementeffective courses of action, and evaluate results.Learning Goal 4: Articulate and Apply a Public Service PerspectiveLearning Objectives4.1 Understand how the values of diversity, equity, integrity, ethical conduct, efficiency,effectiveness, and professionalism shape the formulation and implementation of public policy.4.2 Understand how changing social demographics in the population (including aging,socioeconomic class, race, ethnicity, and language) change the way we govern and how toidentify policies to accommodate these changes.4.3 Understand how to incorporate professional codes of ethics in public service decisionsLearning Goal 5: Communicate and Interact Productively within a Diverse and ChangingWorkforce and CitizenryLearning Objectives5.1 Organize and communicate information to a diverse and changing workforce and citizenry bymeans of oral presentations, written documents and digital media.Additional Learning GoalsAs recommended by NASPAA, in addition to the above five goals, two additional goals relatingspecifically to the John Jay College MPA mission have been formulated. The objectivesassociated with these goals are reflected in the curriculum for the MPA-PPA and the MPA-IO(see goals 6 & 7 in the curriculum maps in Appendices A and B).3. Five-Year Cycle of Assessment ActivitiesFive-Year Cycle of Assessment ActivitiesThe overall goal of the MPA assessment process is to determine, on a systematic and ongoingbasis, how well the curriculum and program structure are working as evidenced by studentachievement of the learning objectives and to use the results to recalibrate the program toimprove student learning. As mentioned earlier, for the first three years of the cycle, our5
assessment population will be all students enrolled for the MPA degree. In the fourth year, wewill begin to look at learning objectives specific to each of the two MPA programsPhase I: Learning Objectives Assessed for All Students Enrolled for the MPA DegreeYear 1: 2010-2011Semester 1: We will define MPA learning objectives in the context of the NASPAA universaldomains (goals) and the two MPA program-specific domains.Semester 2: We will assess student progress toward meeting Learning Objective 3.1:Demonstrate reading, writing and analytical skills necessary for decision-making. We havechosen this objective as the first focus of our student learning assessment due to its fundamentalimportance for student success in both MPA programs. The objective is embedded in severalcourses including PAD 700, the introductory MPA course (see curriculum maps in Appendix A).Students take the MPA Qualifying Examination (MPAQE) near the end of PAD 700 (seeAppendix B). Students failing to take PAD 700 and the Qualifying Exam in a timely fashion aresubject to programming restrictions and may be barred from registering for additional courses.By assessing student achievement when they are enrolled in PAD 700, we will have a baselinefor comparison with subsequent measures of progress toward meeting learning objective 3.1.A scoring rubric for the MPAQE will be used to obtain a numerical score that will be the metricfor student progress in achieving Learning Objective 3.1. Student scores will be calculatedaccording to four levels of performance – exceeds expectations, meets expectations (two levels),and does not meet expectations (see Appendix C for description of the MPAQE and the rubricand scoring system used).Year 2: 2011-2012Semester 1: As an indirect metric, we will analyze student responses to an on-line survey askingthem to assess how well they think they have progressed in meeting learning goals 1-3 andlearning goal 5. The survey was sent via email to all students in the MPA program using SurveyMonkey in 2011.Semester 2: Assess MPA learning goals 1-3 and 5Learning Goal 1: Lead and Manage in Public GovernanceLearning Goal 2: Participate in and Contribute to the Policy ProcessLearning Goal 3: Analyze, Synthesize, Think Critically, Solve Problems and MakeDecisionsLearning Goal 5: Communicate and interact productively within a diverse andchanging workforce and citizenry.6
We will use a rubric that has been designed to assess projects completed in PAD 771, the MPACapstone course that must be passed by all MPA students in order for them to graduate (seeAppendix D for description of Capstone). The rubric, which will be used to assess spring 2012capstone projects, will align with the four learning goals listed above. Using this rubric, scoreswill be calculated for all student capstone projects. We will use the scores to assess studentprogress in achieving learning goals 1- 3 and learning goal 5 according to four levels ofperformance – exceeds expectations, meets expectations (2 levels) and does not meetexpectations.We will also assess student progress on elements in the grading rubric for the spring 2012Capstone project that are identical to the elements in the fall 2011 MPAQE rubric used to assesslearning objective 3.1 Demonstrate reading, writing and analytical skills necessary fordecision-making to identify differences between students who are completing the MPAcompared with those in the early stage of the program.Year 3: 2012-2013Semester 1: Assess learning objectives 1.4, 2.1, 2.2, 2.4, 3.2, 4.1 and 5.1 in addition to thegeneral MPA learning goals.1.4 Understand how governments finance public projects and programs2.1 Apply different policy process models to various management functions including IT,budgeting, HR, and strategic planning.2.2 Understand the influence of politics on policy choices and the practices of policy analysis.2.4 Plan a research or policy analysis project that makes appropriate use of empirical researchtools such as data collection, sampling and statistical analysis.3.2 Define and diagnose decision situations, collect and analyze data, develop and implementeffective courses of action, and evaluate results.4.1 Understand how the values of diversity, equity, integrity, ethical conduct, efficiency,effectiveness, and professionalism shape the formulation and implementation of public policy.5.1 Organize and communicate information to a diverse and changing workforce and citizenryby means of oral presentations, written documents and digital media.The grading rubric for the Capstone project will be redesigned so that we can assess studentprogress in meeting each of these specific learning objectives in addition to the more generalMPA learning goals. Using the rubric, scores will be calculated for each objective for all fall2012 student capstone projects according to four levels of performance – exceeds expectations,meets expectations (2 levels) and does not meet expectations.We will also review fall 2012 syllabi of all core courses to ensure that program and courselearning objectives are being covered by the intended assignments and activities across coursesin the curriculum maps (see Appendices A and B).Semester 2: Using a rubric (to be created), we will use a sample of essay exams/research papersfrom spring 2013 PAD 702 to assess learning objectives 4.1 Identify how the values of diversity,equity, integrity, ethical conduct, efficiency, effectiveness, and professionalism shape the7
formulation and implementation of public policy; and 4.2 Understand how changing socialdemographics in the population (including aging, socioeconomic class, race, ethnicity, andlanguage) change the way we govern and how to identify policies to accommodate thesechanges. The assessment will be made for both learning objectives according to four levels ofperformance – exceeds expectations, meets expectations (2 levels) and does not meetexpectationsBecause we believe that reading, writing and analytical skills are critical for student success, wewill again assess Capstone scores on elements identical to those used in the MPAQE rubric tocontinue our assessment of objective 3.1. Demonstrate reading, writing and analytical skillsnecessary for decision-making. The assessment will be made using spring 2013 capstoneprojects for all students according to four levels of performance – exceeds expectations, meetsexpectations (2 levels) and does not meet expectations. This assessment will be compared withthe MPAQE assessment to determine progress of students who are completing the MPAcompared with those in the early stages of the program.Phase II: Learning Objectives Assessed for the MPA-PPA and MPA-IO ProgramsYear 4: 2013-2014Semester 1: Using a rubric (to be created), we will use a sample of essay exams/research papersfrom PAD 704 (MPA-PPA) and PAD 740 (MPA-IO) to assess learning objective 2.3Understand the influence of economics on policy choices and the practices of policy analysis.The assessment for fall 2013 will be made according to four levels of performance – exceedsexpectations, meets expectations (2 levels) and does not meet expectationsWe will again analyze student Capstone scores on elements identical to those in the MPAQErubric to continue our assessment of student progress in meeting objective 3.1. Demonstratereading, writing and analytical skills necessary for decision-making.Semester 2: For the MPA-PPA, using a rubric (to be created) , we will use a sample of essayexam answers/research papers in PAD 739 to assess learning objective 6.1 Demonstrate theability to apply professional knowledge and skills in public administration. The assessment willbe made according to four levels of performance – exceeds expectations, meets expectations (2levels) and does not meet expectations.For the MPA-IO, using a rubric (to be created) we will use a sample of essay exams/researchpapers in PAD 740 to assess learning objective 6.5 Apply core technical skills for inspectionand oversight such as report writing, and case sampling, and analysis and performancemeasurement. The assessment will be made according to four levels of performance – exceedsexpectations, meets expectations (2 levels) and does not meet expectationsYear 5: 2014-20158
Semester 1: For the MPA-PPA, using a rubric (to be created) we will use a sample of essayexams/research papers from PAD 719 to assess learning objective 7.2 Understand the policymaking process and administration in criminal justice agencies. The assessment will be madeaccording to four levels of performance – exceeds expectations, meets expectations (2 levels) anddoes not meet expectations.For the MPA-IO, using a rubric (to be created) we will use a sample of essay exams/researchpapers from PAD 754 to assess learning objective 7.1 Demonstrate ability to investigatecriminal and managerial fraud, waste, abuse and corruption involving financial records andassets. The assessment will be made according to four levels of performance – exceedsexpectations, meets expectations (2 levels) and does not meet expectations.Semester 2: We will again analyze student Capstone scores on elements identical to those in theMPAQE rubric to continue our assessment of student progress in meeting objective 3.1.Demonstrate reading, writing and analytical skills necessary for decision-making.We will also engage in a comprehensive updating and analysis of all learning objectives. We willmake changes to the curriculum for both MPA programs based on our review ofrecommendations and recalibration of learning objectives.9
APPENDICESAppendix A: Curriculum Maps for MPA-PPAAppendix B: Curriculum Map for MPA-IOAppendix C: The MPA Qualifying Exam (MPAQE)Appendix D: The MPA Capstone10
Appendix A: Curriculum Map for the MPA-PPAGoals and Objectives for the MPA-PPAMethodsCourseOptionsCore n Courses (see key)745747770XXXXXXC CRJA7 7 76 3 10 0 9EMHR711760763XXX703707XXMO712714Goal 1: Lead and Manage in Public Governance1.1 Acquire the knowledge, skills and competencies of leadership and management in public governance and administration, includingthe organization, management, assessment and oversight of human, financial and technical resources.1.2 Understand the core functions, issues, and theories of human resources management.1.3 Understand the major theories of organizational structure and organizational behavior and the preconditions necessary to organizeeffectively.1.4 Understand how governments finance public projects and programs.XXXXXXXGoal 2: Participate in and Contribute to the Policy Process2.1 Apply different policy process models to various management functions including IT, budgeting, HR, and strategic planning.2.2 Understand the influence of politics on policy choices and the practices of policy analysis.2.3 Understand the influence of economics on policy choices and the practices of policy analysis.2.4 Plan a research or policy analysis project that makes appropriate use of empirical research tools such as data collection, sampling andstatistical analysis.XXXXXXXXXXXXXGoal 3: Analyze, Synthesize, Think Critically, Solve Problems and Make Decisions3.1 Demonstrate reading, writing and analytical skills necessary for decision-making.3.2 Define and diagnose decision situations, collect and analyze data, develop and implement effective courses of action, and evaluateresults.XXXXXXXXXXXGoal 4: Articulate and Apply a Public Service Perspective4.1 Understand how the values of diversity, equity, integrity, ethical conduct, efficiency, effectiveness, and professionalism shape theformulation and implementation of public policy.4.2 Understand how changing social demographics in the population (including aging, socioeconomic class, race, ethnicity, andlanguage) change the way we govern and how to identify policies to accommodate these changes.4.3 Understand how to incorporate professional codes of ethics in public service decisionsXXXXXXGoal 5:Communicate and Interact Productively within a Diverse and Changing Workforce and Citizenry5.1 Organize and communicate information to a diverse and changing workforce and citizenry by means of oral presentations, writtendocuments and digital media.Goal 6: MPA-PPA: Apply Professional and Political Knowledge and Skills to Public Administration and Policy Analysis6.1 Demonstrate the ability to apply professional knowledge and skills in public administration.XGoal 7: Mission Specialization Objectives7.1 Court Administration: Understand the decision-making process in court administration.7.2 Criminal Justice Policy and Administration: Understand the policy-making process and administration in criminal justice agencies.7.3 Emergency Management: Be familiar with concepts and technologies of business continuity planning, building design issues andterrorism.X7.4 Human Resources Management: Understand supervisory and administrative responsibilities involving personnel management.7.5 Law and Public Management: Understand the influence of law on public policy and administration7.6 Management and Operations: Understand how to assume supervisory and managerial responsibilities in operational services.CA – Court AdministrationEM – Emergency ManagementMO-Management OperationsCRJ – Criminal Justice AdministrationXXXX XHR – Human Resources11
Appendix B: Curriculum Map for the MPA-IOGoals and Objectives for the 2758715713Specialization Courses (see key)FPAO745747770XXXXXX701749XXOAM7 71 44 5IIO7 71 78 2IOI7 70 51 4FAA704A705Goal 1: Lead and Manage in Public Governance1.1 Acquire the knowledge, skills and competencies of leadership and management in public governance and administration, including theorganization, management, assessment and oversight of human, financial and technical resources.1.2 Understand the core functions, issues, and theories of human resources management.1.3 Understand the major theories of organizational structure and organizational behavior and the preconditions necessary to organizeeffectively.1.4 Understand how governments finance public projects and programs.XXXXXGoal 2: Participate in and Contribute to the Policy Process2.1 Apply different policy process models to, various management functions including IT, budgeting, HR, and strategic planning.2.2 Understand the influence of politics on policy choices and the practices of policy analysis.2.3 Understand the influence of economics on policy choices and the practices of policy analysis.2.4 Plan a research or policy analysis project that makes appropriate use of empirical research tools such as data collection, sampling andstatistical analysis.XXXXXXXXGoal 3: Analyze, Synthesize, Think Critically, Solve Problems and Make Decisions3.1 Demonstrate reading, writing and analytical skills necessary for decision-making.3.2 Define and diagnose decision situations, collect and analyze data, develop and implement effective courses of action, and evaluateresults.XXXXGoal 4: Articulate and Apply a Public Service Perspective4.1 Understand how the values of diversity, equity, integrity, ethical conduct, efficiency, effectiveness, and professionalism shape theformulation and implementation of public policy.4.2 Understand how changing social demographics in the population (including aging, socioeconomic class, race, ethnicity, and language)change the way we govern and how to identify policies to accommodate these changes.4.3 Understand how to incorporate professional codes of ethics in public service decisionsGoal 5: Communicate and Interact Productively within a Diverse and Changing Workforce and Citizenry5.1 Organize and communicate information to a diverse and changing workforce and citizenry by means of oral presentations, writtendocuments and digital media.XXXXXXXXXGoal 6: Apply the Skills and Perspectives of Inspection and Oversight6.1 Demonstrate knowledge of OIG statutory requirements and applicable directives, rules, and regulations.6.2 Demonstrate familiarity with the common structures of organizations, programs, activities, and functions for Federal, state and localinspection and oversight, and the entities, groups, and individuals that interact with inspection and oversight organizations.6.3 Audit the efficiency, economy, and effectiveness of program performance.6.4 Describe how government policies, requirements, and guidelines affect inspection and oversight.6.5 Apply core technical skills for inspection and oversight such as report writing, and case sampling and analysis and performancemeasurement.XXXXXXXXGoal 7: Mission Specialization Objectives7.1 Fiscal Policy Analysis and Oversight: Demonstrate ability to investigate criminal and managerial fraud, waste, abuse and corruptionXX12
Goals and Objectives for the MPA-IOMethodsCourseOptionsCore700involving financial records and assets.7.2 Organizational Assessment and Monitoring: Understand how to assess, evaluate and monitor the performance of public and not-forprofit agencies and private organizations regulated by government.7.3 International Inspection and Oversight: Understand how inspection and oversight functions are implemented in international andmulti-national contexts.7.4 Investigation and Operational Inspection: Understand how to investigate criminal and managerial fraud, waste, abuse and corruption.7.5 Forensic Accounting: Understand Advanced Accounting Concepts and Practices in Forensic AccountingFPAO- Fiscal Policy Analysis and Oversight OAM –Organizational Assessment and MonitoringIIO-International Inspection and ation Courses (see key)FPAO770701749OAM7 71 44 5XIIO7 71 78 2IOI – Investigation and Operational InspectionFAA704A705XXXXXIOI7 70 51 4XXXFA – Forensic Accounting13
Appendix C: The MPA Qualifying Exam (MPAQE)The MPAQE is a two-hour examination in which students answer an essay question based on a journal articletaken from a peer-reviewed public administration journal such as the Public Administration Review. Samplequestions and the grading instrument are provided in advance. Students are given approximately 45 minutes toread the article and conceptualize the response to the essay and an additional 45-75 minutes to write the essay.The question is designed to assess the ability of students to:· Demonstrate reading comprehension;· Demonstrate comprehension of the terms and concepts in the article;· Critically assess what the article says;· Take a position and defend it with a logical argument;· Use references from the article to illustrate and support the position; and· Present an organized and coherently written essay.No student is exempt from the MPAQE. Any student who was exempted from PAD 700 must still sit for theexam. Students who are not enrolled in a current section of PAD 700, or are retaking the MPAQE after afailure, should contact the MPAQE Coordinator for seating in a PAD 700 section that fits the student's schedule.AssessmentThe examination is blind-graded by MPA faculty members. Assessment is completed using the MPAQEGrading Instrument shown below.MPAQE Scaling for AssessmentScaleItemsReadingResponsiveness, Key Terms(Column 1&2)Organization, Vocabulary,Grammar (Columns 3,6 & 7)Argumentation, References(Columns 4 & ectations200-1516-2930-3940 0-78-1314-1718
11
Appendix D: MPA CapstoneThe purpose of the PAD 771 Capstone seminar is for students to demonstrate that they have successfullyachieved the objectives of the MPA program. The student develops a term project on a policy, oversight orpublic management problem by engaging in one of three t
Master of Public Administration (MPA) Assessment Plan Directors Dr. Marilyn Rubin MPA-PP A Dr. Ned Benton MPA-IO March 2012 . 2 1. Outcomes Assessment Philosophy a. Chronology of MPA Assessment Activities Assessment of the John Jay College MPA program has been evolving for close to two decades, reflecting, to a large degree, accreditation .
