Table of Contents
Chapters
- Promoting Dialogue
- Supporting the Client’s Role
- Improving Providers’ Performance
- Table: Activities to Improve CPI
- Box: Supervisors Can Improve CPI
- Best Practices in Training
- Evaluating the Quality of CPI
- Step 1. Engage Stakeholders
- Step 2. Describe Program Goals
- Step 3. Focus the Evaluation Design
- Step 4. Gather Credible Evidence
- Step 5. Justify Conclusions
- Step 6. Ensure Use and Share Lessons Learned
- Moving Beyond Family Planning
- Bibliography
This issue was prepared in collaboration with the Maximizing Access and Quality (MAQ) Initiative of the United States Agency for International Development's Office of Population and Reproductive Health. The MAQ Initiative supports research and evidence-based interventions to promote access and quality of reproductive health and family planning services.
Published by the Information & Knowledge for Optimal Health (INFO) Project, Center for Communication Programs, The Johns Hopkins University Bloomberg School of Public Health, 111 Market Place, Suite 310, Baltimore, Maryland 21202, USA.
Volume XXXI, Number 4,
Fall 2003
Series Q, Number 1
Maximizing Access to Quality
Step 4. Gather Credible Evidence
The credibility of evaluation findings depends on the kinds of data collected, their quality and quantity, and the methods used to collect the data (27). Two issues are crucial: translating CPI concepts into measurable indicators and choosing appropriate sources of data (92).
Indicators must be specific and measurable. Assessing the quality of CPI requires translating general objectives, such as establishing rapport or giving clients their choice, into specific, measurable indicators of an individual’s or a program’s performance. Credible indicators also reflect reliable and objective information and are sensitive to changes in performance.
Creating measurable indicators that accurately assess interpersonal communication and counseling has proved difficult (140). Many indicators require subjective judgment—for example, deciding when a provider has sufficiently explored a family planning client’s needs and preferences. Others, such as listening attentively, may be hard to rate because they are nonverbal or vary across cultures. Even seemingly clear-cut indicators, such as the accuracy of information given, can be surprisingly difficult to assess in the context of actual conversation.
Many organizations have developed extensive lists of CPI indicators as part of training evaluations (42) or facility assessments (7). Table 2 presents CPI indicators from the Quick Investigation of Quality (QIQ), a practical, low-cost, and well-tested tool for measuring the quality of care in family planning services (104, 154).
Multiple data sources strengthen conclusions. Observations of consultations, interviews with clients and providers, and facility audits are common sources of data for evaluating interpersonal communication and counseling. Frequently, CPI evaluations draw on multiple sources of data so that they can offset each other’s weaknesses and so that the consistency of the results can be tested (119).
Approaches to observation have varying advantages and drawbacks. Three approaches have been used for observing client-provider interactions: direct observation, interaction analysis, and simulated clients. Direct observation calls for trained observers—often field workers or supervisors—to watch the consultation and assess the quality of the interaction using a structured observation guide (7, 68). The presence of an observer, however, is the method’s greatest drawback. When clients know they are being observed, they may be less likely to speak, while providers may try harder than usual. Proper selection and training of observers are key to the reliability of the data; otherwise, different observers may interpret and report providers’ actions differently (14).
Interaction analysis of audiotapes or videotapes may avoid the potential bias caused by an observer’s presence, but it costs more (73). After recording and sometimes transcribing the consultation, researchers use a coding guide to classify and quantify everything said by both client and provider (36). Self-assessment tools that help providers assess their own performance, based either on memory or on a tape recording, are a variation on interaction analysis (68, 77).
The simulated, or “mystery,” client approach trains people from the community to recognize what constitutes good care and then sends them to seek health services without revealing to the provider that they are participating in a study. The simulated client reports back to researchers on her experience with the provider and the facility. Carefully defining the profile of the simulated client and limiting reports to specific provider behaviors and facility characteristics increases the objectivity and reliability of reports (96, 120). Of course, this approach cannot assess clients’ behavior.
Interviews elicit clients’ and providers’ opinions. Interviewing clients immediately after their consultations can reveal what took place and whether clients left satisfied. Clients may not accurately remember what happened during the counseling session, however, or they may confuse it with group talks and other clinic events (14). More importantly, a combination of low expectations, courtesy bias (telling interviewers what they want to hear), and fear of criticizing people in authority encourages clients to report complete satisfaction with services and to avoid criticizing the clinic and its staff—thus skewing results in a positive direction (14, 36, 104). Evaluators can reduce this bias by conducting interviews in private (56), away from the facility, using personnel who do not work at the clinic involved (104).
Structured or semistructured interviews with providers can assess their knowledge of family planning, interpersonal communication, and counseling. Interviews may be especially useful for probing providers’ attitudes toward contraceptive use among groups such as adolescents, single women, men, sex workers, women without children, or women with many children. Providers’ actual practices, however, do not always match their knowledge or their professed attitudes (116).
Facility audits assess the setting for CPI. Observers can make site visits to gauge a facility’s readiness to provide quality services, including CPI. For example, observers can assess whether the layout of the building permits privacy for consultations, whether CPI guidelines and job aids are in place, and whether logistical systems assure a steady supply of contraceptives (104).
