Health Information and Publications Network (HIPNET) Home Page

Fully capturing the use of information products and services can be a challenge. While it is fairly feasible to track the reach of information products and services and even to assess how useful they are judged to be, it can sometimes be more difficult to gauge the application of information in short- or long-term user behavior, in programs, or in policies. Dissemination to an intended audience does not guarantee that information will be appreciated or applied. Similarly, even if information users indicate that they have learned something (see Indicator 15, p. 27), the timing and frequency of its application in a practical setting may be hard to observe (NCDDR, 1996; Malchup, 1993).

Use of information can be categorized as instrumental, conceptual, and symbolic. Instrumental use relates to use of information for a particular purpose, conceptual use describes use of information for general enlightenment, and symbolic use refers to information use for the purpose of supporting a predetermined position (Lavis et al., 2003). In this guide we use the general term “use” to encompass all three types of information use. As part of a specific evaluation effort, an evaluation team may decide to better understand the nature of information use and so examine particular types of use—instrumental, conceptual, or symbolic.

Intention to use precedes use. Both are influenced by perception of usefulness (see Indicators 11–16, pp. 25–29). Measuring intention to use is important because it gives an indication of future use. When potential users are first exposed to a product or service, they may have a plan to use the product or service in the future, but they may not have done so yet.

Information can be used exactly as originally produced or can be adapted to fit users’ needs and environmental context. Rogers notes that adaptation (also referred to as reinvention) can occur in order to simplify a complex innovation or to encourage local customization and application. Such reinvention can speed up adoption and enhance the sustainability of an innovation (Rogers, 2003).

To find out about use of information and outcomes stemming from use of information, a researcher can ask users or observe their actions. Asking those who have been exposed to information if they have applied the information products or services, how they have applied them, and what affect they have had is relatively straightforward. While courtesy bias or recall bias may be problems, in some cases the reported use or its result can be verified objectively.

In contrast, observing use of information and outcomes related to use is much more challenging. Determining what information resources were factors in generating a change in behavior or an improvement in clinical practice is difficult. Finding the causal chain and isolating the effect of a specific information resource usually are not practical. Indeed, the person or group taking the action (for example, a policymaker) may not know the source of information acted upon (as when an aide or a lobbyist argues for a proposed policy using information from the resource under evaluation). An exception is the case in which some unique “tag” (for example, a slogan or an acronym) from the information resource shows up as an integral part of the change in policy or practice. Sometimes such a tag can be designed into an information product or service expressly to make its effects easier to track.

Data Requirements: Self-report from audience members on intention to implement changes in behavior or practice based on information from an information product or service, including identification of the product or service and the purpose, scope, and nature of the intended application.

Purpose and Issues: This indicator helps determine if recipients of a product or service plan to make use of its content in the future. In ongoing monitoring, it may be possible to check back with respondents later to find out if plans have been carried out.

Definition: “Adaptation” means the original information product or service has been altered to meet the needs of users in their context. Adaptation might entail translation or simply inserting locally used phrases and terminology or modifying artwork, or it could involve changing the product to take account of local policy, resource availability, and cultural norms. Adaptations also include new (expanded or updated) editions, abridgments, modules for training, modification to address another topic, and transfer to another medium, when these actions are taken by organizations or people other than the original publisher.

Data Requirements: Self-report from users regarding adaptation, including identification of product or service adapted, purpose, extent, and nature of adaptation, outcomes resulting from adaptation (if known).

Data Source(s): User survey (online, email, telephone); letters, email, or other communication requesting permission to create a derivative work; copies of adapted work; requests for technical assistance or funding for adaptation.

Purpose and Issues: This indicator gauges the extended life and increased relevance that an information resource may gain when adapted to meet local needs. In fact, research shows that guidelines, for example, are more effective when they account for local circumstances (NHS Centre for Reviews and Dissemination, 1999). Documenting adaptations is valuable, but it is not possible to know whether one has the complete tally of adaptations. A user may adapt a publication without notifying the original authors or publisher. When adaptations are undertaken independently of the original publisher, they constitute evidence of the adaptors’ judgment that the product will be useful enough in their setting to merit the effort and cost involved in adaptation and publication.

Example:

In Pathfinder’s Packard ASRH project, the Module 16 (Reproductive Health Services for Adolescents) curriculum was adapted by users in all four participating countries. The Philippines and Pakistan changed the content of the curriculum, including adapting case studies to the local context. Ethiopia translated the curriculum into Amharic and cultural adaptations were made by CORHA/Ethiopia, the local partner. Sudan translated the curriculum into Arabic and adapted it to the country context.

Definition: This broad indicator relates to use, and, where known, outcomes of use, of information products or services. “Use” refers to what is done with knowledge gained from an information product or service to change or enhance policies, programmatic or practice guidance, procedures, products, or research methods. Information products (which may include tools, protocols, procedures, manuals, software, systems, methodologies, guides, curricula, indices, key actionable findings) may be used to develop better-informed policies and to improve practice guidelines, program design and management, or curricula, resulting in higher-quality service delivery, more efficient programs, better training and education, or stronger research designs.

Data Requirements: Description of information product or service used, approximate timeframe of use, organization(s) involved, how programs or practice benefited from applying the information, further outcomes associated with use (if known).

Data Source(s): User surveys (online, mail, telephone), usually distributed after the product has been disseminated; informal (unsolicited) feedback; in-depth interviews (phone or in person); copies of policies, guidelines, or protocols referencing or incorporating information from products or services.

Purpose and Issues: The purpose of this indicator is to track how information has been specifically used to inform policy and advocacy or enhance practice, programs, training, education, or research. A difficulty with measuring use is that users may not recall which particular information product was used and how it contributed to a specific outcome.

Research has found that the information contained in guidelines is more likely to be adopted if it is disseminated through educational or training interventions (NHS Centre for Reviews and Dissemination, 1999). A resulting difficulty of measuring the effect of information products (such as guidelines) is separating the effect of the training from that of the information product or service. At the same time, where training and information resources are both necessary components of the trainee’s education or where training is necessary to use an information resource, then the training and information resource together constitute a package that should be evaluated together.

The value of unsolicited, anecdotal reports of use should not be dismissed, given the inherent difficulty in capturing and quantifying use and outcomes of use of information products and services. To that end, it is useful to collect unsolicited feedback from users of products or services, including improvements or achievements based on using a product or service, as well as any problems with using it.

Below are examples of use and, in some cases, outcomes associated with that use. An organization could count the “number” of instances that the use of an information product or service informed policy or advocacy or enhanced practice, programs, training, education, or research. Alternatively, an organization could conduct a survey (see Appendix 6) and report on the percentage of respondents who said they used the information product or service.

Policy example:

Haiti’s National HIV/AIDS Strategy, officially adopted in December 2001, used epidemiological projections prepared by POLICY using the AIDS Impact Model (AIM). In April 2002, when the Minister of Public Health and Population officially released the strategy, the technical group agreed that POLICY’s projections were to be considered the official source of data on HIV/AIDS in Haiti, while awaiting the results of field research. First Lady Mildred T. Aristide also used POLICY’s epidemiological projections in a national message in observance of the International AIDS Candlelight Memorial on May 19. That message was reprinted in the Haitian National Newspaper “Le Nouvelliste.” The AIM projections also were used in the successful proposal to the Global Fund to Fight AIDS. In addition, the National AIDS Strategic Plan, also adopted in December 2001, used demographic and epidemiological projections prepared with POLICY assistance as well as results from the AIDS Program Effort Index for Haiti (POLICY Project, 2002).

Program example:

One of the tools Horizons developed for a study that examined methods for reducing stigma and discrimination in hospitals in India, the “PLHA-friendly Checklist,” was endorsed by the National AIDS Control Organization (NACO) for use in public hospitals and was disseminated to all State Control Societies in the country, the Employees State Insurance Corporation hospitals, and the European Commission-funded HIV/STI Prevention and Care Program in India.

Education example:

Two issues of Population Reports, “Why Family Planning Matters,”(Series J, No 49, July 1999) and “Birth Spacing: Three to Five Saves Lives,”(Series L, No 13, November 2002), have been used each year in a class on Maternal and Child Health in Developing Countries at the University of Washington International Health Program.

Training example:

Fonseca-Becker and colleagues (2002) conducted a study in Guatemala examining the “Synergy of Training and Access to Information in Public Hospitals in Guatemala.” Using a case-comparison approach with 87 reproductive health care providers in 12 hospitals, the team introduced a number of reference materials (The Essentials of Contraceptive Technology handbook and accompanying wall chart and relevant issues of Population Reports) through a training course. The study team found that there was a synergistic effect of training and reference materials on changes in both individual and institutional practice, particularly when an authority figure—the instructor—took part in the diffusion of information, compared with dissemination of the materials independent of the training.

Research example:

New Standard Operating Procedures for research were developed in five countries—Guatemala, Malawi, Panama, Peru, and Zambia—as a result of training for members of ethics committees in those countries using FHI’s Research Ethics Training Curriculum.

Definition: This indicator measures use and the outcomes of the use of information products at the individual level. The types of questions associated with this indicator would include: “Based on something you learned in this publication, have you made any changes in the way you counsel clients?” and “Have you changed the way you do X?” (Such questions could also be rephrased and asked of a supervisor about her subordinates to gauge performance improvement, from the supervisor’s perspective, based on information products.)

Data Requirements: Description of information product or service used, approximate timeframe of use, organization(s) involved, title or position of person(s) involved, how users benefited from applying the product or service, description of context of use, scope of application, further outcomes associated with use (if known).

Data Source(s): User survey (online, in-person, or telephone) conducted as part of product distribution or after a product has been disseminated; in-depth interviews; informal (unsolicited) feedback.Performance checklists (such as clinical skills checklists) and pre- and post-tests can also measure improvements before and after learning from use of information products.

Purpose and Issues: This indicator measures how information products have an effect on users’ knowledge and, ultimately, improved their practice. As with the previous indicator, users may not be able to recall just which information products or services influenced their practice.

Evaluators can easily ask those exposed to a product or information whether and how it affected their practice or for examples of how they put it to use. Such a research approach yields both quantitative information (e.g., percentage of readers who changed their practice) and anecdotal information (what changes did respondents make) that may be quantifiable if there are enough responses to justify categorization. As with policy and program uses, it is helpful to capture feedback from individual practitioners regarding how they used the information product or service.

Example:

“After using the Decision-Making Tool for Family Planning Providers and Clients, there was a significant shift from provider-dominated to shared decision making. During the baseline round, providers were solely responsible for decisions (e.g., to adopt or switch methods) in 44% of sessions and largely responsible in the remaining 56%. After the intervention, providers were largely responsible for the decision in only 19% of sessions and shared the decision with clients in 81% of sessions.” (Kim et al., 2005)

USE INDICATORS
No.
23	Number/percentage of users intending to use an information product or service
24	Number/percentage of users adapting information products or services
25	Number/percentage of users using an information product or service to inform policy and advocacy or to enhance programs, training, education, or research
26	Number/percentage of users using an information product or service to improve their own practice or performance

HIPNET - Health Information and Publications Network

Part IV: Indicators That Measure Use