This article originally appeared as part of the ReShape Newsletter: Sexual Assault Program Evaluation Coalitions Take the Lead (Winter 2013).
An Interview with Emily Tamas, Program Coordinator, Kentucky Association of Sexual Assault Programs (KASAP) and conducted by Kendra Malone, Information & Technical Assistance Specialist, Iowa Coalition Against Sexual Assault December 19th, 2012
1. Why evaluate programs?
State regulation requires that “a center shall evaluate, annually, the level of client satisfaction with its services, using information provided by clients.” In addition to basic regulatory requirements, KASAP’s 2007 strategic plan, developed by our board comprised of the directors of our programs, included the goal to move to quality improvement beyond basic compliance measures. This discussion had the programs themselves determine that they wanted to focus on best practices and to develop standards of excellence.
2. How is your coalition evaluating programs?
To that end, KASAP completes annual compliance monitoring of the programs. This component of evaluation includes a quality section which was added in response to the strategic planning. This is an ongoing and growing process. Client feedback has been an important part of evaluation for years, but we had struggled to find the best format to gain this information. Our current ‘Healing Voices Project’ was developed to capture client experiences of our services in a way that supports the work of healing. The instrument was developed in consultation with Stephanie Townsend, PhD.
3. What tips do you have for working with consultants or researchers?
It is really important to have a contract with a clear scope of work. Before contacting someone our Executive Director checks references and she contacts other coalitions to ask for their recommendations. It is important to do background work before approaching a consultant or researcher.
4. How do you determine outcome measures?
Particularly when programs may have multiple funders and stakeholders to report to who maintain different criteria. All decisions regarding which measures to use are ultimately decided by the KASAP board. Our programs do have multiple quarterly and 6-month reports to complete for various funders. These tend to focus on demographic data and service numbers. Whenever we look at adding measures, there is a discussion about avoiding redundancy as much as possible. We strive to use information we already have and then to add to it in ways that don’t place unnecessary burden on our service providers.
5. How does the coalition support programs in making time for evaluation?
As mentioned before, the programs are part of the decision making process which ensures that we get input from the very folks who will be implementing any decisions made. In the case of the Healing Voices Project, the advocacy checklist forms that advocates complete were designed to take no longer than 5 minutes. Client surveys were developed to require little from the counselor/therapists or session time. All data entry and data analysis for the Healing Voices Project is done by the coalition. KASAP provides an annual report to programs that includes program-specific data and statewide data. This year, for the first time, the coalition will be collecting additional stakeholder information for the programs. In conjunction with the monitoring process, community partners will be surveyed statewide and regional data regarding awareness of and satisfaction with the rape crisis centers will be compiled by the coalition for the programs.
6. What do you do with evaluation results?
KASAP shares the data with the programs annually. Data is reviewed at the state and regional level to further guide strategic planning and policy development.
7. What are the benefits of evaluation?
In addition to helping us maintain focus on continuous quality improvement of all client services, the data supports the current work. Data collected from counseling client surveys provides evidence that RCC counseling contributes to a decrease in trauma symptoms and an increase in client empowerment. This data supports discussions with legislators regarding the funding of programs.
8. How do you quell fears regarding if evaluations show something needs improvement?
KASAP strives to address all needs as an opportunity for improvement. Because of the board involvement in decision-making and our relationship with our programs, there is no “gotcha” approach to identified needs. The coalition works to assist and support programs in development of improvement strategies. The consistency of this approach and the board’s dedication to ongoing improvement keeps the focus on doing what is best for the clients.
9. Other information to add?
There is one central person who handles all evaluation data at the coalition which makes it easier to capture patterns and trends. The evaluation is continuously used by membership. The advocacy checklist is always done when providing accompaniment services. The goal is to have evaluation be an ongoing process in which the coalition will be able to create cumulative reports. Once the coalition has five years worth of information they hope to be able to present a good overview of service provision in conjunction with quarterly reports to guide programming. In terms of how evaluations are given to clients, somewhere between the 1st and 4th session with a client they are asked to give feedback regarding the services they received. The evaluation is given both to better understand their service experience and to also act as an empowerment-building mechanism for the survivor. Counselors have full discretion in regards to determining if a client is able to provide feedback. The ultimate goal of evaluation is to collect client feedback so that the information is useful and palatable to programs.