Transcription
Indiana University School of Social WorkPRAC Annual Report2011-2012 Academic YearIntroductionThe Indiana University School of Social Work (IUSSW) was founded in 1911, and recently celebrated itscentennial anniversary. The School currently offers social work education and the Baccalaureate,Master, and Doctoral level and is one of the few remaining system schools at Indiana University. TheBachelor (BSW) and the Master of Social Work (MSW) programs are both accredited by the Council ofSocial Work Education (CSWE). During the spring semester of 2012, the BSW and MSW programscompleted their self-study for reaffirmation by CSWE under a new curriculum policy statement thatfocuses on competency-based education. A site visit is scheduled for November 2012. The nationalaccreditation covers all the programs of the system school. The BSW program is offered in Indianapolis,Bloomington, Richmond, Gary, and has recently been approved for IU South Bend. The MSW program isoffered in Indianapolis, Richmond, Fort Wayne, South Bend, Gary as well as a one-time cohort insouthern Indiana at IU Southeast which began in January 2012. We are planning to implement a fullyonline MSW program, MSW Direct, in December 2012.Since July 2007, the Division of Labor Studies merged with the School of Social Work and is now anundergraduate program within IUSSW. Labor Studies is also a system-wide program with offices in FieldInstructor, Fort Wayne, IUPUI, Kokomo, IU Northwest and IU South Bend. The program offers aBachelor of Science, an Associate of Science, a Certificate, and a minor in Labor Studies.In order to make the transition to competency-based education as articulated in the CSWE 2008Educational Policy and Accreditation Standards (EPAS), the faculty in the social work programs has beenworking to operationalize the 10 identified core competencies for the various program levels andadvanced graduate concentrations of the IUSSW social work program. Competencies have been linkedto the objectives of each course in the curriculum and will be assessed in the field practica by agencybased associate faculty as well as by the classroom faculty. This report will summarize these efforts byprogram level.
Bachelor of Social WorkThe BSW program operates in multiple contexts that both guide the development and implementationof curriculum as well as provide a framework for assessment of student achievement. Given that theprogram has an upcoming reaffirmation of national accreditation, the first context to be considered isthe shift to competency-based education required by CSWE. During the academic year 2009-2010, theBSW committee reviewed the core competencies mandated by CSWE and identified where in thecurriculum that content is delivered to facilitate student achievement of those competencies. Inaddition to the core competencies, CSWE has identified 41 foundational practice behaviors forgeneralist social work practice (See Appendix A). The BSW committee adopted these 41 practicebehaviors as the operationalization of these competencies. CSWE has also mandated that these practicebehaviors be assessed by two measures, one of which must be in the field practicum; field education hasbeen identified as the signature pedagogy for social work education.The second context for the assessment of social work education is at the level of the IUPUI campus,which is also facing an accreditation process in Fall of 2012. The BSW program has been activelyinvolved in the identification of the Principles of Undergraduate Education (PUL) as major and moderateemphasis in each of the social work courses offered. It is interesting to note that there is significantoverlap between the identified competencies of CSWE for social work education and the PULs:examples include critical thinking, values and ethics and understanding culture and society. To theextent of this overlap, efforts to assess the competencies triangulate the assessment of the PULS andthe existing assessment data on the competencies have implications for the assessment of the PULS.The report below will be focused around the two efforts identified above: A. Assessment ofCompetency-based education in field practicum and through coursework collected in an ePortfolio); andB. PUL assessment.A1. Assessment of Competency-based Education – Field Education1.What general outcome are you seeking?The BSW program seeks to have our 85% of our graduating seniors achieve competency asdemonstrated in their field practicum and ePortfolio on 100% of the 41 identified practicebehaviors adopted as outcomes for the BSW program. Students will also self-assess theircompetency at the benchmark of 85%.
2.How would you know the practice behaviors if you saw them?The practice behaviors were articulated by the Council on Social Work Education and weredesigned to be focused on observable behavior that would be assessed in the field practicaas well as in one other means as identified by the program. The field assessment tool, theLearning Evaluation Tool (LET), has been linked to the 41 practice behaviors with theexpectation that students and agency-based field instructors will identify tasks that studentsmay perform in the agency which will allow the demonstration of each practice behavior.Students are expected to provide documentation of each of the practice behaviors and boththe student and the agency-based field instructor will assess the level of competency.For the second measure of competency used to triangulate the data collected by the FieldInstructors, the BSW program committee voted to adopt the ePortfolio. Products thatdemonstrate each of the 41 practice behaviors were uploaded into the ePortfolio andevaluated by the graduating Senior students’ assigned faculty liaison during the Fall 2011,and evaluation was completed in Spring 2012.3. What opportunities do students have to learn it?The BSW program has 13 required courses, not including practica, which deliver content andopportunities for application of content to prepare students for practice. Each course hasarticulated objectives which have been systematically linked to the CSWE corecompetencies to create an educational matrix. The BSW committee reviewed each syllabiand the designated linkages between competencies/practice behaviors and courseobjectives during the Fall 2011 retreat. A copy of the matrix is included in Appendix D.4.How are you measuring each of the desired behaviors identified in #2 above?Each of the 41 practice behaviors was assessed by both the student themselves and theirfield instructors during their Senior practicum, S482 during Fall 2011. The LearningEvaluation Tool (LET) was developed to provide a tool to gather this data. Each practicebehavior will be assessed using a 7-point scale with 7 being a “Distinguished”, 5 being“Proficient”, 3 being “Apprentice” and 1 being “Not Demonstrated”. This tool is also used toassign a grade for the S482 course, which is either “Satisfactory” or “Fail”.
In addition, faculty members evaluate the same practice behaviors using the studentproducts uploaded into the ePortfolio, using the same scale as above. It is the goal of theBSW committee to create specific rubrics for each of the practice behaviors but that againdid not happen during this period.5.What are the assessment findings?For each of the practice behaviors, we have 3 data points on the level of competency: 1 –ePortfolio as evaluated by the faculty member; 2- Student self-report from the fieldpracticum and 3- Field instructor (agency-based professional social worker) from the fieldpracticum. We identified scores of 5 and above as being “competent” and calculated thepercentage of students who were competent, using the benchmark for success at 85%.Therefore, the scores relate to what percentage of students were rated 5 or above, for eachof the 41 practice behaviors, with our goal being at least 85%.Scores for student final field evaluations were uploaded and analyzed by program staff and63 of 71 final field evaluations were available for analysis (89%). This was an increase fromlast year (62% of students and 20% of Field Instructors) as we hand-collected the data ratherthan asking them to upload it themselves into an online survey collector. Using data fromthe S482 LETs, Field Instructors rated 98% to 100% of students as “competent” on theidentified practice behaviors. Student self-evaluations trended to be slightly lower thanthose of the Field Instructors, however, 95% to 100% of students rated themselves“competent” on their practice behaviors. These scores exceed our identified benchmark of85% for all practice behaviors.For the ePortfolio, the percent participation of students uploading materials ranged amongthe 41 items, from 35 to 54 (49% to 76%). This is also an increase in response rate from theprevious year (25% to 40%) which was a goal for our assessment project. We continue tostrive for 100% participation from students in this important process and have expanded theacademic credit to facilitate student and faculty time to allow this to happen. However,students were rated “competent” on only 19 of 41 practice behaviors or 46%, clearly lowerthan last year’s results.
The results which indicated competency were clustered around social work identity,advancing human rights and social justice as well as assessment which could be consideredstrengths of the current curriculum. There were challenges with critical thinking, research,human behavior and the social environment, contexts and intervention which could beconsidered curricular areas where additional work is needed.There are a number of interpretations that may be made of these data, including majorlimitations to the validity and reliability which are discussed below. The ePortfolio data isNOT consistent with the student self-evaluation or Field Instructor evaluation of thestudents’ practice behaviors. It is possible that classroom instructors have higher standardsfor evaluating students and increased familiarity with the new competency-based systemthan the Field Instructors. The faculty instructors were encouraged NOT to just giveeveryone a 5 (competent) if the student product did not demonstrate competent practice.A document summarizing the data with is presented in Appendix C. The program achievedour benchmark of 85% on all three measures for 19 of 41 practice behaviors (46%). For the23 practice behaviors which did not meet benchmark, only one of three scores did not reachthe benchmark and many (13) just missed the benchmark (80% or above). Generallyspeaking, these results indicate that the program is doing a reasonably good job of achievingthe identified outcomes of the BSW program. However, there were 4 scores in the 60%range that need additional study: PB # 10, PB # 22, PB #36 and PB #41. These four willreceive further attention during the instructor orientation for the seminar class to clarifywhat behaviors are being assessed and possible products which would provide evidence forthat behavior, as well as determine if course content needs to be strengthened.These results are limited by many factors. Both students and agency-based field instructorscontinue to struggle with some of the changes and higher expectations for evaluation. Wechanged the method of hand-loading data from the student LET evaluations which increasedthe response rate for that data. We also provided additional training and support for facultyinstructors to encourage students to upload products for the ePortfolio, as well as addingparticipation points in the course for doing so. In our curriculum revisions, we have addedadditional course credit for the field seminars so students and instructors are truly givencredit for the time they need to upload and evaluate the products.
It is probably not surprising that as the response rate went up, and a broader range ofstudents uploaded products to the ePortfolio, that there were a broader range of scoreswith additional students NOT meeting the 85% benchmark. We were pretty sure last yearthat those who uploaded products tended to be our strongest students! It seems thatstudents are struggling to delineate the actual behaviors that they must demonstrate foreach practice behavior. We believe that as the students become more familiar with thecompetencies and practice behaviors from better integration in EACH social work course,they will do a better job selecting products that truly reflect their competency.We continue to know that without specific rubrics to calibrate measurement of studentproducts in the ePortfolio as well as in the field, there is likely to be a wide range of what isconsidered “competent” among evaluators.6. What improvements have been made based on the assessment findings?After these finding were presented to the BSW curriculum committee in Spring 2012, it wasagain decided that it would not be appropriate to make significant changes to curriculumbased upon them due to the limitations discussed above. The general picture of theprogram was quite positive so there were few clear areas where change seemed to beneeded. As the Indiana University School of Social Work is a system school, data waspresented across all campuses and the IUPUI ePortfolio data was the only area withconsistent concerns. Findings using different methods the previous years had identified thepotential for change necessary in our research sequence, however, that was not supportedby this data from all campuses. However, it WAS identified again for IUPUI students and theprogram director will be convening the faculty who teach in the research sequence toreview curriculum and see whether we might find ways to better prepare students in theresearch area.After working to change the field assessment tools and develop the ePortfolio, we madesome changes in our plans for collecting and evaluating data. The program provided dataentry to ALL the LET field tools and increased the response rate on this data to 89%. Weprovided training to students and field instructors on how to effectively utilize the tools tocomprehensively assess student progress. For the ePortfolio, we began collecting productsin the Spring 2011 Junior practicum which provided a two-semester time period to complete
the ePortfolio. We gave credit for class participation in the Junior practicum for uploadingproducts. In addition, we added the Presentation Maker function to the ePort site, whichwould allow students a specific benefit to them in using the ePortfolio in their job search orgraduate application process, although only a few students took advantage of trainingoffered to allow them to do this.During the coming year, we are reviewing whether all of the 41 behaviors are necessary andappropriate as feedback from students, faculty and field instructors is that they may berepetitive and too difficult to manage for learning as well as assessment. At the same time,it has been identified that the practice behaviors may need to be reworded to be moreobservable and individual rubrics need to be developed to determine what competencelooks like. All of these issues will be discussed during the BSW curriculum committee thisyear.B. PUL Assessment1. What general outcome are you seeking?As one of many undergraduate majors at the IUPUI campus, the BSW program hasidentified how our current curriculum provides opportunities for students todemonstrate their achievement of the PULs. As a result of the educationalopportunities provided in BSW coursework, 80% of Freshman and Sophomore studentswill achieve competency on the major and moderate emphasis PULs in each of theircourses, and 85% (reduced from 90% last year to provide consistency with thecompetency analysis discussed earlier) of Junior and Senior students will achievecompetency on the major and moderate emphasis PULs in each of their courses.**Please note that scores for the small number of Labor Studies students are included inthe IUPUI PUL analysis.2.How would you know it if you saw it?The stated PULs are to be assessed across the BSW curriculum. Faculty members,including associate faculty, have previously been offered a workshop in assessing thePULs are part of their academic responsibilities in teaching BSW courses. This occurredin Fall semesters 2009 and 2010 (it was not offered in Fall 2011 due to emphasis on our
disciplinary accreditation process – it is planned again for Fall 2012). In addition, thereare opportunities for faculty to consult with the program director to enable them tomake good assessments of the PULs based on classroom assignments.3.What opportunities do students have to learn it?Administrative faculty and staff have developed a matrix that identifies major andmoderate emphasis of each PUL in the required BSW program courses. It has beendetermined that these courses provide educational content and experiences that allowstudents to build competency on those identified PULs. The matrix has been reviewedto ensure that all PULs are covered at some point in the BSW professional curriculumand we know that in areas that have less emphasis in the BSW curriculum (e.g. PUL #1aand b), this is additionally covered in their general and supportive educationalrequirements for the BSW degree.4.How are you measuring each of the desired behaviors listed in #2?An evaluation plan has been developed which gathered data on student achievement ofidentified PULs from faculty in identified courses. A summary report was produced bythe IUPUI Office of Information Management and Institutional Research. Data collectiontook place in Spring, 2010, Fall 2010 and Spring/Fall 2011. Faculty members identifiedone (or more) student products from the course which provided the opportunity toassess the identified PUL, for both major and moderate emphasis. These products willbe evaluated according to a 4-point scale, with 3 or above being considered“competent”.5.What are the assessment findings?The School of Social Work received a report from the Office of Information Managementand Institutional Research dated June 2012 which provided faculty ratings from bothSocial Work and Labor Studies from data collected in Spring/Fall 2010, Spring/Fall 2011and Spring 2012. The data was provided for courses at the 100, 200, 300 and 400 levelcourses as well as aggregate scores, for both major and moderate emphasis on thePULs. The tables below summarize the results.
Scores were grouped by 100/200 and 300/400 level courses for analysis. In general,scores averaged at or above the 3.0 level which is the standard of “competent”. Thescores tended to trend higher for 300/400 level courses than in 100/200 coursesalthough it is not possible to tell whether the trends are statistically significant. 80%levels for 100/200 students were achieved for 4 of 12 measures and 85% levels for300/400 students were achieved for 7 of 12 measures (although an additional threescores just missed benchmark at 83%).Some of the findings that seem out of synch with the others seem to be based onsmaller sample which limit interpretation of the data as well as potential problems inthe measurement of the PULs. However, we continue to be concerned that ourstudents are weak in PUL #1a (Written, Oral and Visual Communication) and PUL #2Critical Thinking. These findings are all the more concerning since our students ratedTHEMSELVES as significantly above the IUPUI mean score on Written, oral & visual Skillsand slightly above the mean score on Critical Thinking. The tables and results identifiedabove will be disseminated to all faculty, including associates, teaching in the BSWprogram.
Table 1. PUL analysis for 100 & 200 level coursesPULEmphasisMean %EffectiveTotal Number ofresponses1a. Written, oral 04visual communication2. Critical thinking3. Integration & App.of knowledge4. Intellect. breadth,dep
Bloomington, Richmond, Gary, and has recently been approved for IU South Bend. The MSW program is offered in Indianapolis, Richmond, Fort Wayne, South Bend, Gary as well as a one-time cohort in southern Indiana at IU Southeast which began in January 2012. We are planning to implement a fully online MSW
