Intended Audience

Development of the Guide

Organization of the Guide

Information product. A written or illustrated work made available (published) to more than one person at a time. Information products can be printed or online, CD-ROMs, computer software, audiovisual materials, or recordings (that is, they can be published in any print or electronic medium). Content can take the form of tools, protocols, procedures, manuals, research reports, syntheses of research findings and program experience, methodologies, guides, curricula, meeting and conference summaries and consensus statements, indices, and key actionable findings. They can be sold or distributed free through various channels, including mail, email, and Web sites. (See Appendix 2 for an illustrative list of types of products and publications.)

 

Network. A system of interconnected individuals or organizations that establish and maintain links through which they share information. Networking refers to the process of creating, developing, and maintaining these linkages. A similar, commonly used term for information networks is “knowledge communities.”

 

Reach. The breadth and saturation of product dissemination. It describes the extent to which information is distributed, redistributed, and referred to by organizations and individual users.

 

Service. An effort undertaken by an organization to benefit a defined group or individuals that uses it at their election. Examples of information services include interactive tools such as searchable databases, online libraries, online discussions, and learning programs.

 

Use. What is done with knowledge gained from an information product or service.  It is the way in which information products or services are absorbed and applied to institute or implement changes (NCDDR, 1996; Malchup, 1993).

 

Usefulness. The quality of information products and services that is appropriate, applicable, and practical. Usefulness may include such aspects as user satisfaction, quality, innovation, and relevance.

 

User/audience/recipient. An individual or entity who receives information products and services.

 Framework for Monitoring and Evaluating Health Information Products and Services

---

[1] In December 2005 the Johns Hopkins University Bloomberg School of Public Health, Center for Communication Programs (CCP) developed an operational framework to guide the production and dissemination of its information products. From that operational framework, CCP, with input from members of HIPNET, developed this more generic conceptual framework, which covers the provision of information products and services generally.

 

.

.

The framework groups inputs, processes, outputs (Bertrand and Escudero, 2002), initial outcomes, intermediate outcomes, and intended long-term outcomes (United Way of America, 1996).  The indicators in this guide address reach, usefulness, and use and are discussed below in relation to these framework categories. These categories are defined below (see first figure, p. 5) and then each is discussed further in reference to the second figure (see p. 6):

 

u Inputs: all resources put into a project, such as human and financial capital, equipment, and facilities that enable a program activity to occur.

u Processes: activities undertaken to achieve a specific goal, usually using inputs; refers to how and how well an activity is carried out.

u  Outputs: the products or services resulting from processes at the program level. These outputs reach audience members.

u       Outcomes: the benefits for audiences during or after their exposure to a product, service, or program. Outcomes are at three levels:

·    Initial outcomes: the usefulness of information products and services as determined by audiences’ satisfaction with and perceptions of the quality of products and services.

·    Intermediate outcomes: relate to use or adaptation of information products and services to inform policy, improve programs, enhance training and education, and promote research efforts.

·    Intended long-term outcomes: improvements in health condition or status that may be related to the exposure of health care providers and allied professionals to health information or products.  Programs develop health information products and services bearing in mind how they will contribute to intended long-term outcomes such as improvements in service utilization, health behavior, and health status of the population.

 

Inputs. Projects providing information products and services start with inputs, which can be divided into two main categories: human resources and institutional resources. Human resources are especially important in the provision of information products and services: as the second figure shows (p. 6), knowledge managers (e.g., writers, research analysts, librarians) work with technical experts, researchers, advisory committees, and audience members to develop high-quality products and services. Institutional resources are also a crucial input. They range from “the literature” to management and administration and policy and procedures (e.g., donor and other institutional guidelines) that influence how partners and grantees conduct their work.

 

Processes. A number of important processes take place in the development of information products and services: audience members’ needs and preferences are identified; health information is generated, collected, and organized; and information is synthesized, summarized, and translated. Health information products and services are designed so that they facilitate use and application of their content. For example, publications often advise how evidence-based program approaches can be adopted and adapted to local contexts. Use and application of information products and services is further facilitated by:

·    Illustrating the benefits of emerging evidence-based programs and practice;

·    Relating new policy, programs, and research findings to existing practices;

·    Providing clear steps for application;

·    Providing examples;

·    Facilitating exchange among policymakers, program managers, and researchers of experience with different approaches; and

·    Suggesting simple ways to try these approaches.[1]

 

One key strategy of some information programs is to work together and with others to develop and disseminate technical consensus in a given health area. This process involves bringing technical experts together with the aim of developing guidance that is endorsed and adopted by the group and its collaborating partners. Other programs develop networking approaches to facilitate collaboration and information-sharing. And, of course, information programs develop strategies to disseminate information products and services to key audiences through a number of communication channels, ranging from mailing and shipping to the Internet and CD-ROM.

 

Outputs. The outputs are information products and services that may range from print and electronic publications to various types of information services such as searchable online databases. These products and services are developed to reach one or more of the following audiences—those involved in policy and advocacy, programs and practice, training and education, or research. These key audiences are reached through primary distribution, secondary distribution, and referral. Indicators of reach included in this guide gauge the results of strategic dissemination to the audiences defined in the framework.

 

Outcomes. Outcomes—that is, benefits to the audience—may relate to knowledge, skills, attitudes, behavior, health condition, or health status. For a particular program, outcomes are expected at different levels. For the purposes of this guide, we define three outcome levels: initial, intermediate, and intended long-term outcomes.

 

Initial outcomes. Indicators of initial outcomes capture audiences’ attitudes toward publications and services: Did the audience find the publication or service useful? Usefulness is largely determined by audience satisfaction with content, presentation, and delivery mechanisms and perceived quality of a product or services. More specifically, are audiences satisfied with the extent to which content, presentation, and delivery facilitate understanding, adaptation, adoption, or use? Do users feel that products and services are high-quality (e.g., credible, authoritative, trustworthy, and reputable)? Initial outcomes are comparatively easy to measure, if measurement takes place soon after the audience receives the information products or uses services. Publications and services that the audience deems “useful” are more likely to play a role in policy development or improvements in professional practice.

 

Intermediate outcomes. The perceived usefulness of information products or services in turn translates into intermediate outcomes, which broadly relate to improvements in service delivery. Such improvements resulting from use of information products and services may include enabling quality improvement and increasing access, supporting the scale-up of proven approaches, helping to speed the application of research findings to practice, and contributing to efficiency. Intermediate outcomes are more difficult to evaluate than initial outcomes because users seldom adapt or use information to change the way that they do things (behavior change) unless there are other sources that add to, confirm, or endorse the knowledge presented. Such change in practice is usually the result of multiple factors and thus is difficult to attribute to a specific information product or service.

 

Intended long-term outcomes. Ultimately, information products and services can contribute to intended long-term outcomes, such as increased service utilization, improved health behaviors, and improvements in the health status of the population. This level of outcomes is included to show that information products and services play a supporting role in health outcomes, such as increased service utilization and improved health behaviors and health status. Measuring intended long-term outcomes is beyond the scope of most information projects due to the high resource investment required and the difficulty of attributing such outcomes to exposure to information products and services, since information products and services are often just one aspect of an array of health interventions and a host of environmental influences.

 

To complete the loop, data from routine monitoring and evaluation efforts and occasional, in-depth studies of intermediate outcomes are fed back into inputs, processes, and outputs, informing and improving the development and provision of information products and services. (See lower edge of both graphics.)

  

 

In order to monitor and evaluate information products and services, evaluators may draw on quantitative and qualitative data. Quantitative and qualitative data are complementary, and both can be used to guide program improvements. While quantitative indicators are essential for measuring results and gauging impact (Bertrand and Escudero, 2002), qualitative indicators can provide a more nuanced understanding of results. In this guide some of the quantitative indicators can be enhanced by qualitative data, particularly those in the “Use” section, where indicators focus on documenting use and contextual information around reported use. While it is useful to obtain a sense of how much products and services have been used, it is also helpful to gather additional information related to the context in which the use took place. Such information can be applied to develop broader strategies that encourage application of information. Box 2 describes the information that should be included in a comprehensive description of qualitative results.

 

 Box 2. Writing up a qualitative result

When writing up the achievement of a result, make sure to completely describe what occurred and why it is a result. Apply the basic tenets of good reporting by describing WHO, WHAT, WHERE, WHY, WHEN, and HOW in your description. Make sure that it is clear how your assistance/funding/help contributed to the achievement of the result. The description need not be lengthy, but it should be complete.   Here is general guidance in writing up qualitative results:   ·         Who used information?  For example, who submitted the policy or plan for approval?  Who approved the policy or plan? Who mobilized resources? Who are network members? Who developed or used curricula? ·         What happened? For example, what is the new policy, practice, or approach, and what issues does it address? What amount or kind of resources increased? What advocacy campaigns took place? What kinds of courses were offered? ·         Why is the event important? Describe the significance of the result and include other information as appropriate (for example, the first time an event occurred, a major increase in funding over previous levels, possibilities for future impact, or replication in other areas). ·         Where did the result occur? (Mention country name, region/state, university/training program.) ·         When did this occur?  (Include month and year at a minimum and over what period of time.) ·         How did the result occur? How is the result linked to your efforts and/or financial assistance? (Capture the essence of the work leading up to the achievement of the result.)   Adapted from: POLICY Project. “Project Design, Evaluation, and Quality Assurance Manual.” Washington, D.C., Futures Group, POLICY Project, 2002.

 

 

 

 

Sound research and evaluation require adequate resources—people, time, and funds. Guidelines suggest that M&E costs comprise between 3 and 11 percent of total project funds. In an uncertain and often changing funding environment, programs that provide information products and services are challenged to develop effective information products and services in an efficient manner and to achieve and to demonstrate results with limited resources. As shown in Table 1, routine monitoring activities are relatively low cost, whereas more in-depth research efforts cost more. The table shows the relative cost (low, medium, high) of data collection methodologies that are typically used for M&E of information products and services.