When existing data are not reliable or relevant, managers may set up new data collection systems. To investigate a specific problem, sometimes a quick and informal approach is all that is needed. A manager might respond to a client's complaint by observing, for example, how the receptionist talks with 10 consecutive clients or by asking clients how long they have been waiting. A more formal approach, however, such as the methods described below, is needed for routine monitoring.

Direct observation and simulated clients. Watching consultations and recording the provider's actions on a checklist is a common way to assess counseling and clinical skills. In a comparison of three monitoring methods in Malawi, observations proved to be more reliable than interviewing providers, and observations could collect information on a wider range of activities than client interviews (112). Direct observation has another advantage: Observers can report on both provider's and client's actions and their interaction (190).

When an observer is present, however, staff members are on their best behavior. To overcome this source of bias, some researchers have trained community members to observe providers while pretending to seek care (155, 184, 235, 319, 339). These simulated, or "mystery" clients can be recruited from groups that face special service delivery problems (for example, adolescents or ethnic minorities) in order to explore how providers treat different clients (2, 156). While most often used for evaluations, the simulated client approach also can be used for routine monitoring (232).

Client feedback. As noted (see Chapter 3.2), clients have a valuable perspective on the quality of care. First, as members of the community, clients can define culturally appropriate behavior—for example, whether providers should make eye contact or whether long waits are a burden or a welcome opportunity to socialize. Second, clients generally place more weight on a provider's interpersonal relations, while a trained observer might focus on technical competence. Third, client feedback can indicate whether services have satisfied clients' wants and whether clients are willing and able to carry out decisions made in consultations. To gauge clients' satisfaction, programs can solicit clients' opinions in exit interviews, focus-group discussions, satisfaction surveys, and suggestion boxes (104, 149, 184).

Client exit interviews also can assess provider performance. Studies comparing observations of family planning and health consultations with client exit interviews have found that clients accurately report interpersonal relations as well as concrete actions by providers, such as displaying a flip chart or weighing a child (110, 112, 144).

Clinical audits. Clinical audits are a long-standing approach to ensuring quality of medical care. Conducted by staff members or outside experts, these audits critically review medical records and interview staff about a series of related cases (3, 228). Hospitals in Tanzania and Mozambique have instituted regular audits of maternal and perinatal mortality in which staff openly discuss failures in a supportive and constructive fashion. Death rates have declined as a result (58, 233, 287). The critical approach of clinical audits, however, can demoralize the staff unless properly presented (15). The poor quality of routine medical records in many developing countries also limits clinical audits (144).

Inspections and accreditation visits. Accreditation and monitoring systems generally rely on supervisors or inspection teams who visit a service site and check an extensive list of indicators (15, 137). They review the entire operation of a facility, including support functions as well as client care (15, 258). Inspections can find substandard performance, and they can reward superior care with higher ratings.

A growing number of developing countries are establishing national accreditation programs to manage quality (137, 177, 281, 285). Most accreditation programs work at the hospital level, but the PROQUALI Project has adapted the approach to the clinic level in northeast Brazil (174). State accreditation teams, made up of medical professionals, visit the reproductive health units of participating clinics every 6 to 12 months. Over the first 14 months, accreditation scores at the five pilot clinics improved, on average, from a baseline of 13% of quality criteria met to 94% (5). Accredited clinics—those that score 100%—can display a quality seal and will be promoted incommunity campaigns (176).

Accreditation and inspection programs typically employ outside inspectors who are presumed to be objective and have a broad range of experience and knowledge. They can recognize problems that less knowledgeable insiders may miss. Outside experts may not fully understand the local situation or be able to gain the trust of staff, however. Thus staff members may conceal problems from inspectors and reject their recommendations (15, 149, 228).

An alternative approach is for internal teams of staff members to assess their own facilities (15, 149, 219, 227, 228). Their familiarity with the local situation can help them interpret data, broaden the range of problems identified, and suggest practical solutions (15, 149, 228). Internal assessments also give staff members "ownership" of the results, so that they are more likely to make recommended changes.

Recognizing that both approaches are useful, quality control experts are adding elements of self-assessment to the more conventional inspection approach. A medical monitoring system developed by AVSC International, for example, relies on experienced local physicians not only to inspect service delivery systems during periodic site visits but also to aid the ongoing COPE self-assessment process (15) (see sidebar, Approaches to Quality Improvement).

Peer review and individual self-assessment. Some programs have tried monitoring providers by peer review and self-assessment (378). In Angola, for example, where a long-running civil war prevented regular supervision of nurses at rural family planning clinics, a self-assessment checklist helped maintain the nurses' sense of direction and purpose (60).

These techniques have proved to be both accurate measures of provider performance and effective forms of feedback that can supplement or substitute for outside supervision (378). The Indonesian Midwives Association compared self-assessment with peer review by trained midwives and with outside observation. They found that the three approaches provided comparable information. Feedback from peer reviewers, however, led to substantial gains in performance that were not seen with self-assessment or outside observation (222).

Operations Research. Although Operations Research (OR) is primarily a problem-solving tool rather than a data collection tool, it can generate information within a few weeks or months about how well a service delivery system functions (55, 292, 325). OR typically is used on a one-time basis to find practical solutions to specific service delivery problems. Using a variety of different research techniques, managers or outside researchers assess the current quality of care, diagnose problems, test the feasibility of different approaches to service delivery, and evaluate their impact on the quality of care (29, 55, 343).

Service statistics and Management Information Systems (MIS). Managers can transform routinely collected data on clients into valuable information on quality by totaling the figures each month and calculating simple rates. With the help of summary reports and graphs, they can then analyze trends or compare facilities (65, 196, 364). A sudden change in a service statistic should prompt a search for its causes.

A complete Management Information System (MIS) can be costly to set up and difficult to operate (65), and many existing MIS collect few data that reflect service quality (127, 343). As an alternative, systems are being developed specifically to monitor quality (284). For example, IPPF affiliates in the Dominican Republic and Guatemala have used the computerized Clinic Management System to uncover quality issues such as the short duration of implant use (104) and to identify individual providers with high complication rates or inappropriate method mix (184).

Situation Analysis and the Rapid Evaluation Method. The most widely used approach to assessing the quality of an entire family planning organization is the Situation Analysis, developed and managed by the Population Council (see sidebar, Resources for Quality Control). Since 1989 Situation Analysis has been carried out or planned in 30 countries in Africa, Latin America, Asia, and the Near East (254, 258).

In a Situation Analysis trained research teams collect data from a representative sample of facilities for about six weeks. A standard Situation Analysis includes observations of consultations, interviews with clients and providers, a review of service statistics, and an inventory of equipment and supplies (252, 254).

A Situation Analysis reports on how well a program's subsystems are functioning and makes recommendations. Often, however, program managers have not made systematic efforts to address the weaknesses identified (12, 253). In contrast, the 1991-92 Situation Analysis in Burkina Faso prompted development of long-term national plans for family planning and maternal and child health services, an improved MIS, new training curricula and reference materials on counseling, reorganization of the flow of contraceptive supplies to the regions, and more (13, 286).

Successive Situation Analyses in the same country are beginning to track a program's progress in improving quality of care (253). For example, a comparison of 1989 and 1995 Situation Analyses in Kenya found that contraceptive methods and communication materials had become more widely available and that clients were receiving more information on methods. The number and quality of supervisory visits remained low, however (259).

The World Health Organization (WHO) has developed a flexible approach to assessing the quality of care, staff performance, and client satisfaction, called the Rapid Evaluation Method (REM) (389) (see sidebar, Resources for Quality Control). While the REM and the Situation Analysis collect similar data, the REM has no standardized data collection methodology. Instead, a REM team of decision-makers, managers, trainers, and providers is assembled to direct the assessment. This team decides which issues and service levels to investigate, designs special data collection instruments, and presents its findings to a national or regional seminar that drafts a plan of action.

The REM has been used to evaluate national health programs in five countries. For example, a REM was conducted during the 1991 cholera epidemic in Guatemala to assess the quality of oral rehydration and treatment services. The team identified specific weaknesses—such as providers' errors in determining a child's degree of dehydration and not advising mothers how to feed a sick child. In response, a training workshop was promptly organized to improve providers' skills (142).