Table of Contents
Chapters
- Promoting Dialogue
- Supporting the Client’s Role
- Improving Providers’ Performance
- Table: Activities to Improve CPI
- Box: Supervisors Can Improve CPI
- Best Practices in Training
- Evaluating the Quality of CPI
- Step 1. Engage Stakeholders
- Step 2. Describe Program Goals
- Step 3. Focus the Evaluation Design
- Step 4. Gather Credible Evidence
- Step 5. Justify Conclusions
- Step 6. Ensure Use and Share Lessons Learned
- Moving Beyond Family Planning
- Bibliography
This issue was prepared in collaboration with the Maximizing Access and Quality (MAQ) Initiative of the United States Agency for International Development's Office of Population and Reproductive Health. The MAQ Initiative supports research and evidence-based interventions to promote access and quality of reproductive health and family planning services.
Published by the Information & Knowledge for Optimal Health (INFO) Project, Center for Communication Programs, The Johns Hopkins University Bloomberg School of Public Health, 111 Market Place, Suite 310, Baltimore, Maryland 21202, USA.
Volume XXXI, Number 4,
Fall 2003
Series Q, Number 1
Maximizing Access to Quality
Evaluating the Quality of CPI
Monitoring and evaluation are essential for any effort to create and maintain good communication between health care providers and clients. Without monitoring and evaluation, it is difficult to know how well providers are performing, how satisfied clients are with services, what problems exist, and whether attempted improvements are effective. The evaluation process also spurs the design of innovative approaches to CPI improvement and tests their usefulness.
Depending on its purpose, a CPI evaluation may range from an informal review of practices at a single clinic, conducted by and for frontline staff, to a rigorous assessment of an entire program conducted by outside experts for the benefit of policy-makers and donor organizations. CPI can be the subject of a stand-alone evaluation or part of a broader evaluation.
No matter what approach is used, CPI evaluations are most helpful when they take a broad view and proceed systematically. To capture the many and varied factors that influence the quality and effectiveness of CPI, evaluations should look at more than how the client and provider interact. They also should examine the technical accuracy of information communicated and whether the facility and larger organizational systems support good CPI. Following a systematic process such as the six-step framework shown in Figure 1 ensures that the findings of CPI evaluations will be credible, relevant, and useful (27).
Step 1. Engage Stakeholders
Many people have an interest in how well a family planning program functions, including providers, clients, community members, managers, and policy-makers. Engaging them in the design and implementation of a CPI evaluation helps ensure that the evaluation will reflect varied opinions and address important concerns (5, 149). It also makes it more likely that stakeholders will value the results and act on recommendations (20, 162).
Step 2. Describe Program Goals
Researchers must understand what a program is trying to achieve before they can measure its effectiveness. This requires a thorough program description that, in the case of CPI, first sets out objectives such as informed choice and confidentiality. Then it explains how various program elements (ranging from contraceptive supply and facility layouts to counseling guidelines and supervision systems) are meant to help achieve these goals.
|
Staff members in Ecuador study data about program performance. Involving managers, providers, and a range of stakeholders in a CPI evaluation makes it more likely that all will value the results. |
Step 3. Focus the Evaluation Design
The purpose of an evaluation determines its design and methodology, including what counts as evidence, how data are gathered and by whom, and what claims can be made about the results. Generally, evaluations fall into one of two categories, depending on their intentions (119, 128, 129):
- Formative evaluations identify ways to improve program operations—for example, by assessing the strengths and weaknesses of CPI supervision practices.
- Summative evaluations measure the outcomes or impact of a program—for example, by assessing whether an accreditation system met its objective of improving the quality of CPI.
Formative evaluations tend to use fairly simple, often qualitative research methods and deeply involve stakeholders in design, implementation, and analysis (119, 129). For example, a formative evaluation of the quality of care at the local level in Tanzania employed a highly participatory approach. Fifty local supervisors helped design a quality measuring tool for staff teams, who used it to observe operations in their own facilities and to reach consensus on whether family planning services met pre-established standards. This approach motivated staff members to examine the quality of services honestly and to address service delivery problems (20).
In contrast, summative evaluations require relatively sophisticated research designs and rigorous, quantitative methods because they drive policy and program decisions at the national or international levels (119, 128, 129). For example, a summative evaluation of the impact of radio serials on CPI in rural Nepal required a well-trained team of data collectors, analysts, and statisticians, substantial resources, and complex logistics. The Radio Communication Project conducted baseline and post-intervention surveys of a nationally representative sample of married women; pre- and post-tests of health workers who participated in distance education via radio; three waves of observations and client exit interviews at selected clinics; and an analysis of client flow data from sentinel health posts (153).
