Edison State College Response Report to the Visiting Committee  
Comprehensive Standard 3.3.2: Quality Enhancement Plan  

Recommendation 11: The committee recommends that the institution develop an adequate assessment plan for the QEP.

Edison State College Response:

With input from faculty and administrators who served on the QEP subcommittees, the QEP leadership team designed an assessment and evaluation plan that incorporates direct measures of student learning and indirect measures of student retention, persistence, and graduation rates, student engagement, and the application of strategies by faculty and staff to promote critical thinking and enhance the likelihood of success for first-year students. In response to the On-Site Committee recommendation to narrow the focus of the QEP, the QEP Implementation Team and lead faculty have streamlined the direct measures of assessment to focus on two (rather than four) frameworks: critical thinking and success strategies. In addition, the team has integrated additional qualitative measures to support the quantitative student achievement data. Since the site visit, assessment processes have been developed and implemented.

I. Direct Measures

Student learning in the SLS 1515 Cornerstone Experience course is evaluated through both qualitative and quantitative measures. The QEP Curriculum subcommittee designed two rubrics that are aligned with the student learning outcomes related to critical thinking and success strategies [ 1 ] [ 2 ]. The two rubrics will be used to provide information concerning student performance on three prescribed assignments: the Critical Thinking Journal, the Success Strategies Presentation, and the Final Essay assignment. The assignments, rubrics and assessment timelines were developed through a series of QEP Curriculum Subcommittee meetings during the Fall 2011 semester. Subcommittees included faculty, staff, and administrators. Faculty will apply established rubrics as they score student assignments in all course sections.

On March 3, 2012, ten members of the SLS 1515 faculty participated in a standardization/norming session using the critical thinking rubric. The faculty engaged in a formative assessment of the student’s initial critical thinking journal entries and practiced using the rubric for scoring the entries. In scoring teams of two, faculty compared scores and discussed the manner in which each team member interpreted the rubric. The faculty teams reviewed the rubric criteria to identify any redundant items. They also reviewed the rubric’s levels of performance descriptions to determine if any lacked clarity or if it was difficult to distinguish boundaries between performance levels. All raters took notes as they scored the student artifacts. Additional notes were taken in a follow-up discussion. The notes were transcribed and compiled into a document [ 3 ] Further, the College’s IRPE office analyzed the teams’ scores to establish inter-rater correlations [ 4 ]. The notes and data were shared with the faculty and analyzed during a Community of Practice meeting [ 5 ]. Both the qualitative and quantitative data derived from the standardization session were used by the lead faculty to refine assignments, rubric dimensions, and scoring methodology. Summative data from both rubrics will be collected at the end of the Spring 2012 term.

During the Spring 2012 semester, faculty will submit rubric data (via carbonless copy forms) to the Division of College and Career Readiness assessment clerks, who will enter the data into spreadsheets. The use of an online data management program such as Canvas or TK20 is being reviewed by the QEP Assessment subcommittee for the implementation year 2012-2013. Initial analyses of the rubric data will be conducted by the Office of Institutional Research, Planning and Effectiveness. The analyses will serve as baseline performance measures against which the performance of subsequent cohorts will be compared. The Dean of IRPE will disseminate and discuss the data with the QEP director and lead faculty. The lead faculty will subsequently share the data with all course faculty members during the Community of Practice meetings to inform instruction and to serve as a point of departure for establishing pedagogical and curricular goals. The QEP Director and lead faculty will report actual results and use of results as part of the First Year Experience Unit Plans [ 6 ] [ 7 ]. With input from the SLS 1515 faculty and the QEP Implementation team, the QEP Director and lead faculty will set new achievement goals in each academic year’s unit plan. These performance expectations are aligned with those established within the ESC Student Assessment Committee’s assessment model [ 8 ].

In addition to the measures of achievement provided by the rubric scores, SLS 1515 students will complete a learning readiness indicator instrument and a nationally-normed critical thinking test. Figure 1 demonstrates the alignment among the student learning outcomes and the assessment measures.

Departmental/Unit Outcome Measurement Method/Assessment Tool Outcome-Specific Goal (Performance Expectation)
Critical Thinking: As a result of successful completion of the Cornerstone Experience course, students will be able to: a) Explore how background experiences impact their values and assumptions and explain how they influence personal relationships; b) demonstrate intellectual rigor and problem-solving skills by analyzing and evaluating information, generating ideas, and resolving issues; c) apply intellectual traits, standards, and elements of reasoning in the context of their personal and academic lives. Results of the Critical Thinking Journal assessment scored with the Critical Thinking Rubric By the end of the Spring 2012 semester, 70% of students who complete the course will achieve a 3 (accomplished) or higher on all relevant aspects of the rubric (10% should achieve a 4: exemplary).
Final Essay Assignment scored with Critical Thinking Rubric By the end of the Spring 2012 semester, 70% of students who complete the course will achieve a 3 (accomplished) or higher on all relevant aspects of the rubric (20% should achieve a 4: exemplary).
Scores on the California Critical Thinking Skills Test By the end of the Spring 2012 semester, baseline data will be established for comparison and goal setting for the 2012-2013 academic year.
Success Strategies: As a result of successful completion of the Cornerstone Experience course, students will be able to: a)develop strategies for effective written and verbal communications, use of technology, listening, reading, critical thinking, and reasoning, and b) demonstrate independence self-efficacy through effective personal management, use of college resources and the development of positive relationships with peers, staff, and faculty. Scores on the Smarter Measure Learning Readiness Indicator “personal attribute” items: time management, procrastination, persistence, academic attributes, locus of control, and willingness to ask for help. By the end of the Spring 2012 semester, baseline data will be established for comparison and goal setting for the 2012-2013 academic year.
Success Strategies Presentation rubric By the end of the Spring 2012 semester, 70% of students who complete the course will achieve a 3 (accomplished) or higher on all relevant aspects of the rubric (20% should achieve a 4: exemplary).
Qualitative data from Final Essay assignment Random sample of Final Essay assignments will be analyzed and discussion of success strategies will be coded. The codes will be grouped into concepts and categories that lead faculty will use to describe the success strategies that appear most salient among respondents. The concepts and categories will be used to develop a survey instrument to be used with students in subsequent semesters for self-report of acquisition and application of success strategies.

Figure 1.Direct Measures

Assessment requirements (prescribed assignments)

  1. Critical Thinking Journal
  2. Success Strategies Presentation
  3. California Critical Thinking Skills Test
  4. Smarter Measure Learning Readiness Indicator
  5. Final Essay Assignment

II. Indirect Measures

Indirect measures (see Figure 2) will be administered on a systematic basis. These measures include the Community College Survey of Student Engagement (CCSSE), the Community College Faculty Survey of Student Engagement (CCFSSE), the Survey of Entering Student Engagement (SENSE), and the Student Instructional Report (SIR II). Surveys and assessment measures for extracurricular student engagement activities and faculty/staff professional development  have been developed to provide feedback for planning, program development, and program effectiveness.

Student retention and success rates provide indirect and intermediate measures of the QEP’s impact. To establish baseline information regarding these important data elements, an initial cohort of FTIC students has been established for term-to-term and year-to-year retention; to qualify for this initial cohort, FTIC students must have enrolled during the 2011-2012 academic year and have college placement test scores that indicate a need for college preparatory coursework in at least two areas. Through the five-year implementation phase, new cohorts will be developed based on evolving inclusions criteria. The QEP Director will capture baseline data for each group.

Cohort graduation rates are important indirect measures of student achievement and the QEP’s success. The QEP Director will obtain graduation rates for FTIC students (by level of developmental studies level) who enrolled in 2008-09; these data will be analyzed to establish a baseline, three-year graduation (150% of required time to degree) rate. To establish initial trends in these graduation rates, baseline data will be collected for qualifying FTIC students entering the college during the 2009-2010 and 2010-2011 academic years. Three-year graduation rates for QEP participants will be compared to these initial data points and trends.

  SENSE CCSSE CCFSSE SIR II ESCGS Peer Mentoring Retention
Rates
Persistence
Rates
Transfer of Training Reports Graduation
Rates
Fall 2012 X     X X X     X  
Spring 2013   X   X X X   X X  
Fall 2013 X     X X X X   X  
Spring 2014   X X X X X   X X  
Fall 2014 X     X X X X   X X
Spring 2015   X   X X X   X X X
Fall 2015 X     X X X X   X X
Spring 2016   X X X X X   X X X
Fall 2016 X     X X X X   X X
Spring 2017   X   X X X   X X X

Figure 2.Indirect Measures

The SENSE will provide information on student engagement, ESC intake processes, student support systems, and student learning during the early weeks of the fall semester. The six categories of findings that are associated with the SENSE include Early Connections, High Expectations and Aspirations, Clear Academic Plan and Pathway, Effective Track to College Readiness, Engaged Learning, and Academic and Social Support Network. Each of these areas can be mapped into the four frameworks of the Cornerstone Experience course.

The CCSSE will also provide information on student engagement, ESC intake processes, student support systems, and student learning during the early weeks of the fall semester. Each of the five benchmark areas, i.e. Active and Collaborative Learning, Student Effort, Academic Challenge, Student-Faculty Interaction, and Support for Learners, have been mapped into the four frameworks of SLS 1515 Cornerstone Experience.

SIR II will provide information on instructor communications, faculty/student interaction, supplementary instructional methods, course outcomes, and student effort and involvement. The student is asked to think about each practice as it contributed to his or her learning in the course. The SIR II categories have been mapped to the four frameworks of SLS 1515 Cornerstone Experience.

In addition to the standardized measures described above, qualitative data will be generated through focus group interviews. In April 2012, the IRPE office will select a representative sample of enrolled students across the College’s three campuses and one center. A series of focus group meetings will be led by the First Year Experience Coordinator, SLS 1515 faculty and staff. A standard set of interview questions [ 9 ] will be employed to guide the discussions. With the permission of the students, the focus groups will be recorded to facilitate analysis. Focus group responses related to student satisfaction and engagement will be coded, then grouped into concepts and categories that lead faculty and staff to understanding the elements of the course and extracurricular activities that increased students’ satisfaction and engagement. The concepts and categories will form the basis for the development of a student survey instrument for use in subsequent semesters. Focus group and subsequent survey data will be disseminated to faculty and staff and used to inform course and program improvement.

The QEP Director, QEP co-chairs, and lead faculty will have primary responsibility for ensuring ongoing assessment and evaluation of all components of the QEP and ensuring results are used for planning, improvement, and decision making. The QEP Director, QEP co-chairs and lead faculty worked closely with the Department of Institutional Research, Planning and Effectiveness to develop an annual unit plan that adheres to College policies and procedures. Each of the QEP’s direct and indirect  measures is tied to a unit plan objective [ 10 ].

Four primary measures will be used to evaluate implementation of the faculty and staff professional development program; these include available resources, levels of participation, levels of satisfaction, and evidence regarding the application of learning. Specific indicators will include the number of faculty and staff participants [ 11 ] [ 12 ] [ 13 ].disciplines and campuses represented by participants, costs of facilitating the workshops, including staff time, and workshop evaluation forms whereby participants rate their satisfaction with the training activity [ 14 ]. In order to determine the efficacy of the training, the QEP Implementation team has designed and administered a survey to all Cornerstone Instructor Modules completers [ 15 ]. The survey uses Likert Scale items and open-ended responses to determine perceptions of the usefulness of knowledge and skills learned, and whether knowledge and skills will be applied to teaching and other student interactions. After each administration, the Training and Development Subcommittee will review the survey results to inform revision of the training modules.

Program QEP
Assessment Cycle Five years beginning AY 2012-13
Purpose/Mission Through the full implementation of the Quality Enhancement Plan, Edison State College’s first-time-in-college (FTIC) students will be self-reliant learners imbued with critical thinking skills.
Departmental/Unit Outcome Measurement Method/Assessment Tool Outcome-Specific Goal
(Performance Expectation)
Once fully implemented, the QEP will facilitate an increase in student retention rates, rates of persistence, and graduation rates. Within course completion rate (derived from course grade distributions) Once fully implemented, students will successfully complete the Cornerstone Experience at a rate of 85% with a C or better.
  Term-to-term retention reports (derived from the Banner Student Information System) Using AY 2011-12 baseline data, term-to-term retention will increase by 5% each year.
  • Baseline for students enrolled in two or more developmental studies, AY 11-12 and 12-13
  • Baseline for students enrolled in any developmental studies, AY 13-14 and 14-15
  • Baseline for students without developmental studies, AY 15-16

 

Year-to-year retention reports (derived from the Banner Student Information System)

Using AY 2011-12 baseline data, year-to-year retention will increase by 3% each year.

  • Baseline for students enrolled in two or more developmental studies, AY 11-12 and 12-13

  • Baseline for students enrolled in any developmental studies, AY 13-14 and 14-15

  • Baseline for students without developmental studies, AY 15-16

 

Cohort graduation reports derived through the Banner Student Information System

This analysis will use the cohort graduation rate associated with students that entered ESC as FTIC during AY 10-11.

  • Cohorts from AY11-12 and AY12-13 who graduate within 150% of the expected time required will increase by 10% when compared to the AY 10-11 baseline

  • Cohorts from AY13-14 and AY 14-15 who graduate within 150% of the expected time required will increase by 10% when compared to the AY 10-11 baseline

  • Cohort from AY15-16 who graduate within 150% of the expected time required will increase by 10% when compared to the AY 10-11 baseline

 

Course Outcome items from SIR II: 29, 30, 31, 32, 33 and Student Effort and Involvement items: 34, 35 and 36

Beginning AY 2012-13, faculty results for these items will be at or above the national average.

Through each phase of implementation, the QEP will foster increased rates of student satisfaction and student engagement. The success of this measure will be demonstrated through the quality of student/student, student/faculty, and student/college engagement.

Engaged Learning items from the SENSE:
19a, 19b, 19e, 19g, 19h, 19i, 19j, 19k, 19l, 19m, 19n, 19o, 19q, 20d2, 20f2, and 20h2

Beginning AY 2012-13, there will be a 5% increase in the Engaged Learning benchmark over the previous year’s results.

 

Student-Faculty interactions items from CCSSE: 4k, 4l, 4m, 4n, 4o, and 4q

Beginning AY 2012-13, there will be a 5% increase in the Student-Faculty interactions benchmark over the previous year’s results.

 

 

Faculty/Student Interaction items from SIR II: 11, 12, 13, 14 and 15
Subset of Active and Collaborative Learning items from CCSSE: 4f, 4g, 4h, and 4r

Beginning AY 2012-13, faculty results for these items will be at or above the national average.
Beginning AY 2012-13, there will be a 5% increase in the subset of Active and Collaborative Learning benchmark over the previous year’s results.

 

Qualitative data from focus group responses

Focus group responses will be analyzed and discussion of student satisfaction and engagement will be coded.  The codes will be grouped into concepts and categories that lead faculty and staff to understanding the elements of the course and extracurricular activities that increased students’ satisfaction and engagement. The concepts and categories will be used to develop a student survey instrument for use in subsequent semesters.  Survey data will be used to inform course and program improvement.

As the faculty complete the Cornerstone Experience Instructor professional development  modules, they will apply newly obtained knowledge to their practices to promote critical thinking and enhance the likelihood of success for first-year students.

Academic Challenge items from CCSSE:  4p, 5b, 5c, 5d, 5e, 5f, 6a, 6c, 7, 9a (Fall 2012)

Beginning AY 2012-13, there will be a 5% increase in the Academic Challenge benchmark over the previous year’s results.

 

Professional Development Surveys

Following completion of the professional development modules, 80% of trained faculty will report using critical thinking and first-year student success strategies as measured on Likert scale items.

 

Supplementary Instructional Methods items from SIR II: 22, 23, 24, 25, 26, 27 and 28 and Communication items: 6, 7, 8, 9 and 10

Beginning AY 2012-13, faculty results for these items will be at or above the national average.

As the staff and administrators complete the Cornerstone Experience Services professional development  modules, they will apply practices that promote critical thinking and success to their interactions with first-year students.

Transfer of training staff and administrator self-report instrument

Following completion of the professional development modules, 80% of trained staff and administrators applying critical thinking and first-year student success strategies as measured on Likert scale items.

 

SENSE items from A Plan and a Pathway to Success category:
18d, 18g, 18e, 18f, and 18h

Beginning AY 2012-13, there will be a 5% increase in A Plan and Pathway to Success benchmark over the previous year’s results.

 

Unit plans from administrative and student services areas

Training goals will be reflected in at least one unit plan objective annually from all administrative and student services units.

Figure 3. QEP Program Outcomes and Performance Expectations

Documentation  
Supporting Documentation Description
1. Critical Thinking Rubric QEP Rubric
2. Success Strategies Rubric QEP Rubric
3. Critical Thinking Results Qualitative Data for Student Journal Scoring
4. Critical Thinking Data Inter-Rater Correlation
5. Meeting Minutes Community of Practice QEP Meeting
6. Unit Plan 1548 First Year Experience Unit Plan
7. Unit Plan 1551 First Year Experience Unit Plan
8. Assessment Model Student Assessment Committee (SAC)
9. Focus Group Protocol Guidelines for First Year Experience Focus Groups
10. Unit Plans QEP Unit Plans
11. Cornerstone Training Report Summary of training
12. Faculty Completer Report Cornerstone Training Completion
13. Staff/Administration Completer Report Cornerstone Training Completion
14. Cornerstone Workshop Evaluations Training Evaluation Summary
15. Cornerstone Instructor Workshop Evaluations Training Evaluation Summary